How to add a circuit breaker to an outbound HTTP client#

A dependency stops answering. Every request to it waits for the full timeout, and your worker pool fills with calls that are all going to fail. Requests that have nothing to do with that dependency start queueing behind them, and a partial outage upstream has become a total outage in your service.

What you get

You will end up with a breaker that stops calling a failing dependency and probes it once per cool-off. This is for you if one slow upstream can exhaust the concurrency your whole service shares.

Short answer

Count consecutive failures against a threshold. Reaching it opens the circuit, and calls then fail without being made. After a cool-off, let one request through: a success closes the circuit and a failure reopens it for another cool-off. Derive the half-open state from the clock rather than from a timer, so an idle breaker holds nothing.

You will need

Node 22 or later, and a dependency whose failures last longer than a retry. A breaker is worth adding where a timeout is expensive, which usually means a synchronous call on a request path rather than a background job.

Approaches compared

The in-process breaker and the mesh differ in what they can see. A breaker in your code knows which call failed and why, and it can hold a different circuit per endpoint. A mesh sees connections and statuses, and it protects every language in the fleet without any of them being changed.

Keep three states and one clock

The state machine is small, and the derived half-open state is what keeps it that way.

export function makeBreaker({ threshold = 3, coolOffMs = 5000, now = Date.now } = {}) {
  let state = 'closed'
  let failures = 0
  let openedAt = 0

  return {
    get state() {
      // half-open is derived from the clock rather than set by a timer, so a
      // breaker with no traffic holds no timer and cannot leak one.
      if (state === 'open' && now() - openedAt >= coolOffMs) return 'half-open'
      return state
    },

Injecting now is what makes the cool-off testable without waiting for it. A test moves the clock and asserts the state, so the suite runs in milliseconds and the assertions are exact rather than approximate.

Order matters where a breaker sits beside a timeout and a retry, and the order is fixed by what each one measures. The timeout decides when a single attempt has failed. The retry decides whether to try again. The breaker counts the calls that ended in failure and decides whether to try at all. Put the breaker outermost and it sees one failure per call, which is the number the threshold is written against. The resilience4j documentation describes the same layering for a different runtime. Its default is a percentage over a sliding window, which is the refinement to reach for once a consecutive count proves too blunt. The AWS builders’ library argues the harder half of the question, which is what a service should do with the request it has just refused to send.

Reopen on the first failed probe

The half-open case is the one implementations get wrong. A dependency that is still down should cost one probe, not another threshold’s worth of traffic.

        // A single failure in half-open reopens the circuit for another
        // cool-off. Counting up to the threshold again would send the
        // threshold's worth of traffic at a service that is still down.
        failures = current === 'half-open' ? threshold : failures + 1
        if (failures >= threshold) {
          state = 'open'
          openedAt = now()
        }

Which failures count is the other decision. A 500 from the dependency counts. A 404 does not, because the dependency answered correctly and the resource is missing. Counting 4xx responses opens the circuit for every client that sends a bad request, which turns a caller’s bug into an outage for everybody else.

Check it worked

The demo drives one breaker through failure, cool-off, a failed probe, and recovery, with the clock under its control.

node demo.mjs

t=0ms request 1        closed     -> closed     call failed
t=0ms request 2        closed     -> closed     call failed
t=0ms request 3        closed     -> open       call failed
t=0ms request 4        open       -> open       no call made
t=0ms request 5        open       -> open       no call made
t=5000ms request 6     half-open  -> open       call failed
t=10000ms request 7    half-open  -> closed     ok
t=10000ms request 8    closed     -> closed     ok

Requests four and five are the point of the whole mechanism: they cost nothing, because no call was made. Request six is the probe that found the dependency still down, and it reopened the circuit rather than letting a second probe through.

node --test breaker.test.mjs

1..3
# tests 3
# suites 0
# pass 3
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 109.26619

When it goes wrong

One breaker guards several dependencies. A single instance shared across three APIs opens for all three when one of them fails, and the failure looks like a much wider outage than it is. Keep one breaker per dependency, keyed by host, and per endpoint where one endpoint is much slower than the rest.

The second failure is a breaker that never opens because every call is retried first. Retries happen inside the breaker’s call, so three retries of a failing request count as one failure. Either count each transport attempt, or set the threshold in terms of the calls the breaker actually sees.

The third is a threshold that trips on ordinary noise. Three consecutive failures is a reasonable default on a dependency with steady traffic, and it is far too sensitive on one that is called twice a minute. Where traffic is thin, a percentage over a rolling window fits better than a consecutive count.

When not to do this

Do not open a circuit on a caller’s own bad request. Validation failures and missing resources are answers, and treating them as dependency failures lets one broken client cut off everybody else.

Do not add a breaker without deciding what happens when it is open. A rejection with no fallback is still a failed request, so the value comes from the cached value, the queued job, or the degraded response you serve instead.

Do not put a breaker in front of something you cannot stop calling. A breaker on your only database turns a slow database into an unavailable one, and the fast failure buys nothing when there is no alternative path.

Do not tune the cool-off shorter than a dependency’s restart. A five second probe against a service that takes ninety seconds to come back produces eighteen failed probes. Each one is a request the dependency has to answer while it is starting.

Related how-tos

• Retry fetch calls with exponential backoff in Node.js

• Enforce one total deadline across all retry attempts

Last verified

Verified 2026-09-06 against Node 22.22.2. Both output blocks are what the preceding command printed. The demo drives an injected clock, so the transcript is the same on every run.