Uptime monitoring — what to point at, and what not to

There are three layers of downtime detection, and they answer different questions. Only the third can wake you up.

Layer	Where it runs	Detects	Tells you
Liveness beat	Inside the API	A gap in its own heartbeat	After the outage, as an alert card
HTTP monitor	Outside	The API not answering	During, if you configure it
Dead-man’s-switch	Outside	Our pings stopping	During, even when we’re unreachable

The first ships and runs already (apps/api/alerts/livenessBeat.ts). The other two need a (free) third-party account, which is the only part nobody has done for you.

The trap: /api/health always returns 200

Do not point an uptime monitor at /api/health.

That endpoint governs Traefik routing, so it returns 200 even when the database is down — deliberately. On 2026-06-29 a DB node outage hung it, the container was pulled from the load balancer, and the entire API 503’d instead of degrading. Keeping it 200 is what makes a DB blip a degradation rather than a blackout.

The cost is that a monitor asking “is it 200?” reports green through a total database outage — a false negative on the failure you most want caught.

Point it at /api/health/strict

Same reading, expressed as a status code:

• 200 — database reachable.
• 503 — database unreachable.

Nothing routes on it, so it is free to be strict. systemSchedulesHealthy deliberately does not affect the code: recurring billing failing to register is real and is in the body to keyword-match on, but every request still serves, and paging for it at 03:00 teaches you to silence the pager.

curl -i https://platform-api.viite.ai/api/health/strict

Monitor config (any provider)

Setting	Value
URL	https://platform-api.viite.ai/api/health/strict
Method	GET
Interval	1–5 min
Up when	HTTP 200 and body contains "db":"up"
Down after	2 consecutive failures (avoids paging on one dropped packet)

Keep the keyword check even though the endpoint is strict. It costs nothing and it survives someone later “fixing” the status code back to 200 — belt and braces on the exact mistake this page exists to prevent.

Free tiers that cover this: UptimeRobot (50 monitors, 5-min), Better Stack (10 monitors, 3-min), Healthchecks.io (20 checks).

Why not GitHub Actions

It looks free and is not, on a private repo: Actions bill per job rounded up to the minute, so a 5-minute cron is ~8,640 minutes/month against a 2,000-minute free allowance — roughly $50/month, and it competes with CI for the same allowance. A public repo avoids the cost but makes your operational status public. Actions cron is also best-effort and drifts under load. A free SaaS tier is better on every axis.

The dead-man’s-switch — the only one that beats an unreachable host

An HTTP monitor has to reach us. If the host is up but unreachable — a firewall change, a DNS mistake, an expired certificate — it sees an outage and cannot tell you which. Inverting the direction fixes that: we ping out; the service alerts when the pings stop. Nothing of ours has to be alive for its alarm to fire.

This is already wired into the liveness beat and is dormant until you set one env var.

Better Stack heartbeats (what we run)

Create a heartbeat and copy its ping URL. It takes HEAD, GET or POST; the beat sends a plain GET.

https://uptime.betterstack.com/api/v1/heartbeat/<token>

Setting	Value	Why
Expected every	10 min	The beat runs every 5, so one missed beat is not an incident — two are.
Grace period	5 min	Absorbs a deploy restart without paging. Total 15 min to alarm, matching the in-process downtime threshold.

Treat the ping URL as a credential. Anyone holding it can send heartbeats — masking a real outage — or append /fail to trigger a false page. It lives in Coolify env only: never in git, and never in this repo, because this site is public.

And never put it in a local or worktree .env. This one is not about disclosure, it is a false negative, which is worse: a dev API running on your laptop pings the same heartbeat, so Better Stack stays green while production is down and the alarm never fires. The value belongs in exactly two places — the main checkout’s apps/api/.env, which is the source of truth the sync reads, and Coolify. Worktrees deliberately do not get it; the beat simply stays dormant there, which is the correct local behaviour.

Setting it

HEARTBEAT_PING_URL is declared in three places so a Coolify row actually reaches the container (see adding an env var). Set the value on the platform app in Coolify.

Prefer setting the single row in the Coolify UI over bun run env:coolify:sync --apply, which pushes your whole local env file and can revert prod flags — if you do use the sync, dry-run it first and read the diff.

Adding this env var: the full list

An env var only works in production when it is declared in all of these. The compose entry is the one that silently breaks things if forgotten: a Coolify row with no environment: line never reaches the container, so the feature stays dormant with no error anywhere.

File	Entry
apps/api/core/env/parseEnvSchema.ts	HEARTBEAT_PING_URL: z.string().default('')
.env.example	documented, empty
docker-compose.coolify.yml	- HEARTBEAT_PING_URL=${HEARTBEAT_PING_URL:-}
operations/env-vars.md	inventory row + class

Two properties worth knowing, both deliberate and both tested:

• A database outage does NOT stop the ping. The ping happens before, and outside, the database work. A dead-man’s-switch says the process is alive; a DB outage is a degraded service, not a dead one, and pinging through it stops the loudest alarm we have from crying wolf. The HTTP monitor above is what catches that case.
• A failing ping never breaks the beat. A monitoring provider being down must not take us with it.

Not used yet: /fail

Better Stack accepts <url>/fail (and <url>/<exit-code>) to report a failure explicitly, rather than waiting for pings to stop. That would let the heartbeat alone cover database outages too, collapsing both monitors into one signup.

It is deliberately not wired, because it changes what the alarm means: today the heartbeat says the process is alive and nothing else, and the HTTP monitor owns degradation. Merging them makes one page mean two things. Worth revisiting if the HTTP monitor never gets set up — one imperfect alarm beats a perfect one that does not exist.

Recommended combination

1. HEARTBEAT_PING_URL → a dead-man’s-switch. Catches the API being dead or unreachable, and pages.
2. An HTTP monitor on /api/health/strict. Catches degradation the process itself cannot see — most importantly a database outage while the API is otherwise happily serving.
3. The liveness beat, already running. Catches the gap after the fact and leaves a card explaining what did not run, which is the part you read in the morning.