# How to add health and readiness checks to a worker service > Give a worker with no HTTP port a liveness check that flags a stuck loop, and a readiness check that holds work until its broker and store answer. Source: https://voxgig.com/howto/add-health-checks-to-a-worker-service - Audience: platform-team - Level: beginner - Languages: typescript - Verified: 2026-10-21 - Published: 2026-10-20 ## Short answer Give the worker two Fastify routes for the Kubernetes probes, /livez and /readyz, so a stuck process restarts and a rollout waits for a worker that can work. Liveness asks only whether the consume loop has turned within its stall limit. Readiness asks the broker and the store at the moment it is called, and the worker takes no messages while it fails. A heartbeat row in Postgres covers schedulers that cannot probe. --- ## You will need Node 22 or later, and a worker that consumes messages and serves no HTTP. Verified 2026-10-21 against Node 22.22.0 and Fastify 5.12.7. The sample calls its routes through Fastify's [`inject`](https://fastify.dev/docs/latest/Guides/Testing/), which opens no port, and stands in for the broker and the store with in-memory objects. The heartbeat table runs on [`node:sqlite`](https://nodejs.org/api/sqlite.html) in place of Postgres, and NATS Micro is described from its documentation rather than run. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [A Fastify endpoint](https://fastify.dev/docs/latest/Reference/Routes/) behind [Kubernetes probes](https://kubernetes.io/docs/concepts/workloads/pods/probes/) | The worker runs on Kubernetes, or on any scheduler that polls HTTP | A listening port in a process that had none, and a 200 proves only what the handler checks | Nothing in your platform polls HTTP, or the worker runs outside a cluster | | [A heartbeat row in Postgres](https://www.postgresql.org/docs/current/sql-insert.html) | The worker already writes to Postgres, and something outside the cluster must notice a stall | A write every interval from the loop, a watchdog you run and alert from, and detection no faster than its threshold | The store is the dependency that fails, so the row stops for a reason the watchdog cannot tell apart | | [NATS Micro ping and stats](https://pkg.go.dev/github.com/nats-io/nats.go/micro) | Services built on the micro package, where discovery and per-endpoint counters come with it | Kubernetes can send a NATS request only through an exec probe, and a ping says nothing about the store | The worker uses plain subscriptions, or Kubernetes must decide restarts | The HTTP endpoint is the one every scheduler understands, but it proves only what its handler checks, so the readiness handler has to ask the broker and the store itself. NATS Micro answers a ping over the broker connection, which no HTTP probe can do,[^1] and Kubernetes can send that ping only through an exec probe or a sidecar. A heartbeat row proves the loop finished a turn within the threshold, and the price is a watchdog you run yourself. ## Ask two questions with different consequences Kubernetes acts differently on each probe. It restarts a container that fails liveness. A Pod that fails readiness is removed from the endpoints of every Service that matches it, and a rolling update [keeps the previous Pods](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) until enough replacements are ready.[^2] The [probe documentation](https://kubernetes.io/docs/concepts/workloads/pods/probes/) draws the line between them: liveness passes when the app itself is healthy, and readiness also checks each back-end service the app requires. ```js title="health.mjs" export function createHealth({ checks, now, stallAfterMs }) { let lastTurn = now() return { // The consume loop calls this every time it goes round, including a fetch // that came back empty, so an idle worker on a quiet queue stays alive. turned() { lastTurn = now() }, live() { const idleMs = now() - lastTurn if (idleMs <= stallAfterMs) return { ok: true } return { ok: false, reason: `consume loop has not turned for ${idleMs / 1000} s` } }, async ready() { const failing = [] for (const [name, check] of Object.entries(checks)) { try { if (!(await check())) failing.push(name) } catch { failing.push(name) // a check that throws is a dependency that is not there } } return failing.length ? { ok: false, waiting: failing } : { ok: true } }, } } ``` `live()` looks only at the consume loop, through the time since its last turn. `ready()` runs every check at the moment the probe asks, so it reports each connection as it is, not as a flag remembered from startup. A check that throws counts as a missing dependency rather than crashing the probe. ## Serve both routes on a port of their own The worker serves nothing else over HTTP, so the routes get a small Fastify app on a port that no Service selects. Start it before the broker and the store connect, so the first probe gets an answer. ```js title="server.mjs" export function healthServer(health) { const app = Fastify() app.get('/livez', async (request, reply) => { const result = health.live() return reply.code(result.ok ? 200 : 503).send(result) }) app.get('/readyz', async (request, reply) => { const result = await health.ready() return reply.code(result.ok ? 200 : 503).send(result) }) return app } ``` Point the probes at those routes. Readiness runs more often and gives up sooner, so a lost store connection takes the Pod out within ten seconds, while a restart waits for three failed liveness checks in a row. ```yaml title="probes.yaml" livenessProbe: httpGet: { path: /livez, port: health } periodSeconds: 10 failureThreshold: 3 readinessProbe: httpGet: { path: /readyz, port: health } periodSeconds: 5 failureThreshold: 2 timeoutSeconds: 2 ``` The demo walks one worker through its life and asks both routes at each step. ```bash node demo.mjs ``` ```text output process started livez 200 readyz 503 waiting for broker, store broker connected livez 200 readyz 503 waiting for store store connected livez 200 readyz 200 store connection drops livez 200 readyz 503 waiting for store store reconnected livez 200 readyz 200 consume loop stuck 45 s livez 503 readyz 200 consume loop has not turned for 45 s ``` Readiness follows the connections, and liveness follows the loop. The store outage turns readiness off and leaves liveness on, so Kubernetes holds the Pod out without restarting it, because a restart brings no database back. Restarting every worker during an outage adds load to the recovery, which is the cascading failure the probe documentation warns about. The stuck loop is the reverse case, and a restart is what fixes it. ## Take messages only while readiness passes Readiness alone routes no work away from a worker. Kubernetes uses it to choose Service endpoints, but a queue consumer has no Service in front of it, so the broker keeps delivering to a Pod that failed its probe. The worker has to hold its own consumption on the same checks. ```js title="worker.mjs" export function createWorker({ broker, store, health }) { return { async tick() { const lines = [] if (!(await health.ready()).ok) { health.turned() // waiting for a dependency is a turn: the loop is alive return lines } for (const message of broker.take()) { try { await store.write(message) lines.push(`${message.id} delivery ${message.deliveries} written`) } catch (err) { broker.retry(message) lines.push(`${message.id} delivery ${message.deliveries} failed: ${err.message}`) } health.turned() // each message is progress, so a long batch is not a stall } health.turned() // a fetch that came back empty is a turn too return lines }, } } ``` Each turn asks `ready()` first and takes nothing while it fails. The checks run on every turn, so make each one read the client's own connection state instead of sending a query. A query that hangs during an outage would stall the loop, and liveness would restart a worker that only had to wait. The worker records a turn while it waits, after every message, and after an empty fetch. A worker waiting for its store stays alive, and a long batch on a slow store counts as progress rather than a stall. A tick that throws records nothing, so a fetch that fails on every tick fails liveness, and Kubernetes restarts the Pod with a new broker client. ```bash node pitfall.mjs ``` ```text output readiness that asks the broker and the store 0s readyz 503 waiting for store 1s readyz 503 waiting for store 2s store connected 2s readyz 200 2s order-1 delivery 1 written ``` The order waited at the broker until the store connected, and its first delivery was the one that wrote it. ## Write a heartbeat row for a watchdog A heartbeat row covers what the probes cannot: a scheduler that does not poll HTTP, and a check from outside the cluster that the loop finished a turn. The loop writes one row per worker, and a watchdog selects the rows whose last beat is past its threshold. ```js title="heartbeat.mjs" const BEAT = `INSERT INTO worker_heartbeat (worker_id, beat_at) VALUES (?, ?) ON CONFLICT (worker_id) DO UPDATE SET beat_at = excluded.beat_at` const STALE = `SELECT worker_id, beat_at FROM worker_heartbeat WHERE beat_at < ? ORDER BY worker_id` ``` ```bash node watchdog.mjs ``` ```text output watchdog at 2026-01-01T10:01:00.000Z, threshold 30 s, 3 rows stale worker-b last beat 2026-01-01T10:00:05.000Z 55 s ago stale worker-c last beat 2026-01-01T09:58:40.000Z 140 s ago ``` The upsert keeps one row per worker, which is why four beats left three rows. On Postgres, `beat_at` becomes `timestamptz`, the placeholders become `$1` and `$2`, and the database can supply the time. Choose that clock deliberately, because `now()` means the start of the transaction.[^3] Write the beat from the loop rather than from a timer, since a timer keeps beating while the loop it vouches for is stuck. ## Check it worked The suite pins each behavior the page relies on, from the 503 that names the store to the address the server listens on. ```bash node --test health.test.mjs ``` ```text output 1..10 # tests 10 # suites 0 # pass 10 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 249.798847 ``` The second test is the one to keep: a store outage fails readiness and leaves liveness at `200`, so an outage never turns into a restart. The fifth test holds the worker's own consumption to the same checks at startup, and the sixth does so during an outage. The seventh and eighth tie liveness to progress: five ten-second writes in one batch keep it at `true`, and forty seconds of ticks that throw turn it `false`. ## When it goes wrong Deliveries fail for the first seconds after every deploy. Readiness proves only that the process started, so the worker takes messages before its store connects, and each failure spends a delivery. ```bash node pitfall.mjs ``` ```text output readiness that proves the process started 0s readyz 200 0s order-1 delivery 1 failed: store not connected 1s readyz 200 1s order-1 delivery 2 failed: store not connected 2s store connected 2s readyz 200 2s order-1 delivery 3 written ``` Two deliveries were spent on a store that was not there. With a delivery limit of two, the order would have reached the dead-letter queue before the store came up. Run the checks when asked, and hold consumption on them. Probes fail with connection refused while the routes answer inside the container. Fastify [listens on `localhost`](https://fastify.dev/docs/latest/Reference/Server/) unless told otherwise, and Kubernetes probes the Pod's IP address. Pass `host: '0.0.0.0'` to `listen`, as `server.mjs` does. The worker restarts every few minutes on a quiet queue. Liveness counts only processed messages, so an idle loop looks stuck. Record a turn for every fetch, including one that comes back empty. ## When not to do this Do not put the store check in liveness. A database outage would then restart every worker at once, which adds load to the recovery and brings the database back no sooner. Do not add an HTTP port to a worker that nothing polls. A scheduled job, or a worker on a platform without probes, gets more from the heartbeat row and an alert on its age. Do not rely on NATS Micro alone when Kubernetes decides restarts. Kubernetes cannot send a NATS request without an exec probe, and a ping answered over the broker says nothing about the store. Use its ping and stats for discovery and dashboards, beside the HTTP routes. Do not treat the heartbeat as a liveness probe. The watchdog's threshold is minutes of lost work before anyone hears about it, so it backs up the probes rather than replacing them. ## Related how-tos - [Run service startup steps in a fixed order](/howto/run-service-startup-steps-in-order) - [Limit concurrency in a message worker](/howto/limit-concurrency-in-a-message-worker) - [Make queue consumer writes safe under redelivery](/howto/queue-consumer-writes-under-at-least-once) - [Add a health endpoint agents can poll](/howto/add-a-health-endpoint-for-agents) ## Last verified Verified 2026-10-21 against Node 22.22.0 and Fastify 5.12.7. Every output block is what the command preceding it printed. The probes ran through Fastify's `inject` against in-memory stand-ins for the broker and the store, without a Kubernetes cluster, a NATS server, or Postgres. The heartbeat statements ran on `node:sqlite`, and NATS Micro is described from its documentation. [^1]: Every micro service answers three requests it never registered, `PING`, `INFO`, and `STATS`, on subjects that start with `$SRV`, as the [package documentation](https://pkg.go.dev/github.com/nats-io/nats.go/micro) lists them. A ping sent to `$SRV.PING` with no service name returns a response from each service instance listening, so the request that checks one worker also counts the fleet. [^2]: A rollout whose replacement Pods never pass readiness does not fail so much as stop. The [Deployment documentation](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) allows it ten minutes by default, then adds a condition with the reason `ProgressDeadlineExceeded`. It notes that Kubernetes takes no action on a stalled Deployment other than to report that condition. Under the default limits at least three quarters of the previous workers stay up, which for a queue consumer is the outcome you wanted. [^3]: Postgres keeps several clocks. [`now()`](https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-CURRENT) returns the start of the current transaction, `statement_timestamp()` the start of the current statement, and `clock_timestamp()` the actual current time, which moves even within one statement. A worker that writes its beat with `now()` inside a long transaction records the moment the transaction began, so a busy worker can look stale.