Health Check Endpoints Done Right — Designing /health So It's Worth Monitoring

Every service has a /health endpoint, and most of them are lies. Some return 200 as long as the process can execute a return statement, certifying nothing. Others check the database, the cache, the queue and three third-party APIs on every probe — and become the mechanism by which one slow dependency takes down an entire healthy fleet. A health check is an API contract with your infrastructure: load balancers, orchestrators and monitoring all make decisions based on what it says. It deserves to be designed, not pasted in from the first tutorial that turned up.

Here's how to build health endpoints that tell the truth — and that are safe to point automation at.

Shallow vs deep: two questions, not one

Health checks answer one of two very different questions, and most problems come from mixing them up:

• Shallow (liveness): "Is this process alive and able to serve anything?" No dependencies checked — the process answers, therefore it lives. Cheap, fast, safe to call constantly.
• Deep (readiness/dependency): "Can this service do useful work right now?" Checks the database connection, the cache, the queue, maybe critical third parties. Expensive, slower, and — crucially — its answer depends on things outside this process.

Both are legitimate. The catastrophe is using a deep check where a shallow one belongs.

The cascade failure: deep checks in load-balancer probes

Here's the classic outage. Your load balancer probes each app instance's /health every few seconds and pulls failing instances out of rotation. Someone "improves" /health to also verify the database. Then the database has a slow moment — a lock, a failover, a saturated connection pool.

Every instance's health check now fails simultaneously. The load balancer, doing exactly what it was told, marks every instance unhealthy and takes the whole fleet out of rotation. Users get connection errors — total outage — when the actual condition was "database briefly slow," which the app might have ridden out with a few slow requests and some retries. The health check didn't detect the incident; it manufactured it. Worse, the removed instances stop receiving traffic, their pools drain, the database recovers, and you're now in a thundering-herd restart.

The rule that falls out:

• Load-balancer and orchestrator probes get shallow checks. They answer "should traffic go to this instance?" — and a database problem is not this instance's problem; pulling the instance fixes nothing and amplifies everything.
• Deep checks are for monitoring and alerting — systems that notify humans rather than automatically removing capacity. A human seeing "all instances report DB unreachable" draws the right conclusion; a load balancer can't.

Separate /live from /ready

The cleanest pattern — popularized by container orchestration but useful everywhere — is splitting the two questions into two endpoints:

• /live — shallow. "The process is up." If this fails, the supervisor should restart the process. It should have no dependencies whatsoever; a deadlocked event loop failing to answer is exactly what it exists to catch.
• /ready — moderately deep. "This instance can serve real traffic right now." Fails during startup (warming caches, running migrations), during graceful shutdown, or when an instance-local dependency (its DB connection pool, not the DB itself) is broken. Load balancers route on this.

One instance failing /ready is routine — that's a deploy. All instances failing /ready at once is an alarm that should page someone, and it's a signal your monitoring should be watching for, because your load balancer will respond to it by helpfully serving nothing.

Return real status codes — not 200 with sad JSON

An anti-pattern that never dies:

HTTP/1.1 200 OK

{
  "status": "unhealthy",
  "database": "connection refused"
}

Every piece of infrastructure that consumes health checks — load balancers, orchestrators, uptime monitors — keys off the status code first. A 200 that says "unhealthy" in the body reads as healthy to most of them. Return 200 when healthy and 503 Service Unavailable when not; put detail in the body for humans and richer tools:

HTTP/1.1 503 Service Unavailable

{
  "status": "unhealthy",
  "checks": {
    "database": { "status": "fail", "latency_ms": 5003 },
    "cache":    { "status": "pass", "latency_ms": 2 },
    "queue":    { "status": "pass", "depth": 12 }
  }
}

A monitor asserting status code == 200 and, belt-and-braces, body contains "\"status\": \"pass\"" on the components you care about, now catches both failure shapes. And degraded-but-serving states ("cache down, running slow off the DB") are best expressed as 200 with a "status": "degraded" body — up for the load balancer, alert-worthy for monitoring.

Keep it cheap, honest, and quiet

A few design rules that separate production-grade health endpoints from liabilities:

• Cheap to call. Probes hit this endpoint constantly, from multiple sources. Dependency checks should be trivial (SELECT 1, a cache ping) — never a table scan, never a full ORM boot. Cache sub-check results for a few seconds so ten probes don't mean ten database round-trips.
• Timebox every sub-check. A health check that hangs for 30 seconds because a dependency hangs is itself an outage amplifier. A 1–2 second timeout per sub-check, and the endpoint reports the timeout as a failure instead of inheriting it.
• Skip auth, but don't leak. Probes can't do OAuth dances, so /health is usually unauthenticated — which means it must not expose internals. Connection strings, hostnames, versions and stack traces in a public health endpoint are free reconnaissance. Public gets 200/503 and component pass/fail at most; the verbose diagnostic view lives behind auth or an internal network boundary.
• Don't check third parties you can't act on in the checks that gate traffic. Your payment provider being slow shouldn't mark your instances unhealthy. Watch third parties from your monitoring, as their own tracked dependencies.
• Exclude health-check traffic from your analytics and rate limits — and log it separately, or not at all. Nothing ruins request metrics like a probe every two seconds.

Point external monitoring at them

An endpoint this thoughtful deserves an audience beyond your load balancer. Internal probes see your service from inside the network — external monitoring sees it the way users do, DNS and TLS and routing included, and it keeps watching when your internal tooling is part of the outage. Monitor /ready (or your deep check) externally at a regular interval, assert on the status code and a body keyword, and alert on latency creep as well as hard failures — a health check that took 20 ms all year and takes 900 ms this week is telling you something.

Make /health an asset, not a liability

Shallow checks for machines that remove capacity, deep checks for systems that alert humans, real status codes, tight timeouts, nothing leaked. Do that, and /health becomes the single most monitorable URL you own. CompleteStatus is built to sit on the human-alerting side of that contract — API monitors with status-code, keyword and JSON body assertions, response-time tracking to catch the slow drift, and multi-channel alerting when /ready starts saying no. If you're hardening your stack before the busy end-of-year season, this is one of the highest-leverage afternoons you can spend — start monitoring your health endpoints free, or see the full toolkit on the features page.