How to add health and readiness checks to a worker service#

Your queue worker has no HTTP port, so the cluster cannot ask it anything. A Pod whose store connection never came up counts as running, takes messages, and fails every write. A rollout replaces working Pods with Pods that cannot reach the database. A consume loop stuck on a promise that never settles looks the same as an idle one.

What you get

You will end up with a liveness route that gets a stuck worker restarted, a readiness route that holds consumption and rollouts until the broker and store answer, and a heartbeat watchdog. This is for you if a worker consumes messages and serves no HTTP.

Short answer

Give the worker two Fastify routes for the Kubernetes probes, /livez and /readyz, so a stuck process restarts and a rollout waits for a worker that can work. Liveness asks only whether the consume loop has turned within its stall limit. Readiness asks the broker and the store at the moment it is called, and the worker takes no messages while it fails. A heartbeat row in Postgres covers schedulers that cannot probe.

You will need

Node 22 or later, and a worker that consumes messages and serves no HTTP. Verified 2026-10-21 against Node 22.22.0 and Fastify 5.12.7. The sample calls its routes through Fastify’s inject, which opens no port, and stands in for the broker and the store with in-memory objects. The heartbeat table runs on node:sqlite in place of Postgres, and NATS Micro is described from its documentation rather than run.

Approaches compared

The HTTP endpoint is the one every scheduler understands, but it proves only what its handler checks, so the readiness handler has to ask the broker and the store itself. NATS Micro answers a ping over the broker connection, which no HTTP probe can do,¹ and Kubernetes can send that ping only through an exec probe or a sidecar. A heartbeat row proves the loop finished a turn within the threshold, and the price is a watchdog you run yourself.

Ask two questions with different consequences

Kubernetes acts differently on each probe. It restarts a container that fails liveness. A Pod that fails readiness is removed from the endpoints of every Service that matches it, and a rolling update keeps the previous Pods until enough replacements are ready.² The probe documentation draws the line between them: liveness passes when the app itself is healthy, and readiness also checks each back-end service the app requires.

export function createHealth({ checks, now, stallAfterMs }) {
  let lastTurn = now()
  return {
    // The consume loop calls this every time it goes round, including a fetch
    // that came back empty, so an idle worker on a quiet queue stays alive.
    turned() {
      lastTurn = now()
    },
    live() {
      const idleMs = now() - lastTurn
      if (idleMs <= stallAfterMs) return { ok: true }
      return { ok: false, reason: `consume loop has not turned for ${idleMs / 1000} s` }
    },
    async ready() {
      const failing = []
      for (const [name, check] of Object.entries(checks)) {
        try {
          if (!(await check())) failing.push(name)
        } catch {
          failing.push(name) // a check that throws is a dependency that is not there
        }
      }
      return failing.length ? { ok: false, waiting: failing } : { ok: true }
    },
  }
}

live() looks only at the consume loop, through the time since its last turn. ready() runs every check at the moment the probe asks, so it reports each connection as it is, not as a flag remembered from startup. A check that throws counts as a missing dependency rather than crashing the probe.

Serve both routes on a port of their own

The worker serves nothing else over HTTP, so the routes get a small Fastify app on a port that no Service selects. Start it before the broker and the store connect, so the first probe gets an answer.

export function healthServer(health) {
  const app = Fastify()
  app.get('/livez', async (request, reply) => {
    const result = health.live()
    return reply.code(result.ok ? 200 : 503).send(result)
  })
  app.get('/readyz', async (request, reply) => {
    const result = await health.ready()
    return reply.code(result.ok ? 200 : 503).send(result)
  })
  return app
}

Point the probes at those routes. Readiness runs more often and gives up sooner, so a lost store connection takes the Pod out within ten seconds, while a restart waits for three failed liveness checks in a row.

livenessProbe:
  httpGet: { path: /livez, port: health }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: health }
  periodSeconds: 5
  failureThreshold: 2
  timeoutSeconds: 2

The demo walks one worker through its life and asks both routes at each step.

node demo.mjs

process started          livez 200  readyz 503  waiting for broker, store
broker connected         livez 200  readyz 503  waiting for store
store connected          livez 200  readyz 200
store connection drops   livez 200  readyz 503  waiting for store
store reconnected        livez 200  readyz 200
consume loop stuck 45 s  livez 503  readyz 200  consume loop has not turned for 45 s

Readiness follows the connections, and liveness follows the loop. The store outage turns readiness off and leaves liveness on, so Kubernetes holds the Pod out without restarting it, because a restart brings no database back. Restarting every worker during an outage adds load to the recovery, which is the cascading failure the probe documentation warns about. The stuck loop is the reverse case, and a restart is what fixes it.

Take messages only while readiness passes

Readiness alone routes no work away from a worker. Kubernetes uses it to choose Service endpoints, but a queue consumer has no Service in front of it, so the broker keeps delivering to a Pod that failed its probe. The worker has to hold its own consumption on the same checks.

export function createWorker({ broker, store, health }) {
  return {
    async tick() {
      const lines = []
      if (!(await health.ready()).ok) {
        health.turned() // waiting for a dependency is a turn: the loop is alive
        return lines
      }
      for (const message of broker.take()) {
        try {
          await store.write(message)
          lines.push(`${message.id} delivery ${message.deliveries} written`)
        } catch (err) {
          broker.retry(message)
          lines.push(`${message.id} delivery ${message.deliveries} failed: ${err.message}`)
        }
        health.turned() // each message is progress, so a long batch is not a stall
      }
      health.turned() // a fetch that came back empty is a turn too
      return lines
    },
  }
}

Each turn asks ready() first and takes nothing while it fails. The checks run on every turn, so make each one read the client’s own connection state instead of sending a query. A query that hangs during an outage would stall the loop, and liveness would restart a worker that only had to wait.

The worker records a turn while it waits, after every message, and after an empty fetch. A worker waiting for its store stays alive, and a long batch on a slow store counts as progress rather than a stall. A tick that throws records nothing, so a fetch that fails on every tick fails liveness, and Kubernetes restarts the Pod with a new broker client.

node pitfall.mjs

readiness that asks the broker and the store
  0s  readyz 503 waiting for store
  1s  readyz 503 waiting for store
  2s  store connected
  2s  readyz 200
  2s  order-1 delivery 1 written

The order waited at the broker until the store connected, and its first delivery was the one that wrote it.

Write a heartbeat row for a watchdog

A heartbeat row covers what the probes cannot: a scheduler that does not poll HTTP, and a check from outside the cluster that the loop finished a turn. The loop writes one row per worker, and a watchdog selects the rows whose last beat is past its threshold.

const BEAT = `INSERT INTO worker_heartbeat (worker_id, beat_at) VALUES (?, ?)
  ON CONFLICT (worker_id) DO UPDATE SET beat_at = excluded.beat_at`

const STALE = `SELECT worker_id, beat_at FROM worker_heartbeat
  WHERE beat_at < ? ORDER BY worker_id`

node watchdog.mjs

watchdog at 2026-01-01T10:01:00.000Z, threshold 30 s, 3 rows
  stale  worker-b  last beat 2026-01-01T10:00:05.000Z  55 s ago
  stale  worker-c  last beat 2026-01-01T09:58:40.000Z  140 s ago

The upsert keeps one row per worker, which is why four beats left three rows. On Postgres, beat_at becomes timestamptz, the placeholders become $1 and $2, and the database can supply the time. Choose that clock deliberately, because now() means the start of the transaction.³ Write the beat from the loop rather than from a timer, since a timer keeps beating while the loop it vouches for is stuck.

Check it worked

The suite pins each behavior the page relies on, from the 503 that names the store to the address the server listens on.

node --test health.test.mjs

1..10
# tests 10
# suites 0
# pass 10
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 249.798847

The second test is the one to keep: a store outage fails readiness and leaves liveness at 200, so an outage never turns into a restart. The fifth test holds the worker’s own consumption to the same checks at startup, and the sixth does so during an outage. The seventh and eighth tie liveness to progress: five ten-second writes in one batch keep it at true, and forty seconds of ticks that throw turn it false.

When it goes wrong

Deliveries fail for the first seconds after every deploy. Readiness proves only that the process started, so the worker takes messages before its store connects, and each failure spends a delivery.

node pitfall.mjs

readiness that proves the process started
  0s  readyz 200
  0s  order-1 delivery 1 failed: store not connected
  1s  readyz 200
  1s  order-1 delivery 2 failed: store not connected
  2s  store connected
  2s  readyz 200
  2s  order-1 delivery 3 written

Two deliveries were spent on a store that was not there. With a delivery limit of two, the order would have reached the dead-letter queue before the store came up. Run the checks when asked, and hold consumption on them.

Probes fail with connection refused while the routes answer inside the container. Fastify listens on localhost unless told otherwise, and Kubernetes probes the Pod’s IP address. Pass host: '0.0.0.0' to listen, as server.mjs does.

The worker restarts every few minutes on a quiet queue. Liveness counts only processed messages, so an idle loop looks stuck. Record a turn for every fetch, including one that comes back empty.

When not to do this

Do not put the store check in liveness. A database outage would then restart every worker at once, which adds load to the recovery and brings the database back no sooner.

Do not add an HTTP port to a worker that nothing polls. A scheduled job, or a worker on a platform without probes, gets more from the heartbeat row and an alert on its age.

Do not rely on NATS Micro alone when Kubernetes decides restarts. Kubernetes cannot send a NATS request without an exec probe, and a ping answered over the broker says nothing about the store. Use its ping and stats for discovery and dashboards, beside the HTTP routes.

Do not treat the heartbeat as a liveness probe. The watchdog’s threshold is minutes of lost work before anyone hears about it, so it backs up the probes rather than replacing them.

Related how-tos

• Run service startup steps in a fixed order

• Limit concurrency in a message worker

• Make queue consumer writes safe under redelivery

• Add a health endpoint agents can poll

Last verified

Verified 2026-10-21 against Node 22.22.0 and Fastify 5.12.7. Every output block is what the command preceding it printed. The probes ran through Fastify’s inject against in-memory stand-ins for the broker and the store, without a Kubernetes cluster, a NATS server, or Postgres. The heartbeat statements ran on node:sqlite, and NATS Micro is described from its documentation.