# How to decide which layer retries a failed vendor call

> Give network errors, 429 responses, 5xx, and handler crashes one retry owner each among stripe-node, p-retry, and JetStream, then count vendor calls to prove it.

Source: https://voxgig.com/howto/which-layer-retries-a-vendor-call

- Audience: platform-team
- Level: intermediate
- Languages: typescript
- Verified: 2026-10-21
- Published: 2026-10-20

## Short answer

Give each failure class one retry owner among stripe-node, a p-retry provider, and NATS JetStream, and switch the other two off for it. Let maxNetworkRetries in stripe-node own network errors and 5xx, since it reuses one idempotency key across its attempts. Let p-retry own 429 responses, within a shared retry budget. Let JetStream redeliver only after a crash, and end the message with term after any vendor failure. Derive the idempotency key from the message.

---
## You will need

Node 22.18 or later, which runs TypeScript files without a build step, plus stripe-node and
p-retry. A JetStream consumer whose `ack_wait` and `max_deliver` you control, and an idempotency
key for vendor writes. Verified 2026-10-21 against Node 22.22.0, stripe 23.0.0, and p-retry 8.0.1.
The sample calls a local fake of the Stripe API in place of Stripe, and drives an in-process model
of a JetStream consumer in place of a NATS server. Turning vendor errors into outcomes a caller can
branch on is the subject of
[mapping vendor error responses onto message errors](/howto/map-vendor-errors-onto-message-errors).

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [NATS JetStream redelivery](https://docs.nats.io/learn/jetstream/acknowledgment) | A worker that dies mid-message, where only the bus outlives the process | Re-runs the whole handler, so a create the vendor already accepted runs twice unless its key comes from the message | A failure the handler saw and classified, which it should settle with `term` |
| [p-retry in the provider](https://github.com/sindresorhus/p-retry#readme) | 429 responses, and any class the SDK does not retry, with one budget for the whole process | You write the classification and the budget, and its defaults retry ten times over seventeen minutes | The vendor SDK already retries that class with a stable key |
| [stripe-node maxNetworkRetries](https://github.com/stripe/stripe-node#network-retries) | Network errors, 409 conflicts, and 5xx from Stripe, with one key reused across attempts | One setting covers network errors and 5xx together, and it never retries a plain 429 | Any vendor other than Stripe, or a 429 you want to wait out |

The three layers differ in what they know. stripe-node knows Stripe's errors and keeps one key
across its attempts, but a single setting covers network errors and 5xx together. The provider sees
every call to the vendor, so it can hold a budget for the whole process, at the price of writing
the classification yourself. JetStream outlives a dead worker, and pays for that by re-running
everything the handler does.

## Count what one failure costs

The stacked setup in `stacked.ts` retries at every layer. stripe-node keeps its default of two
retries,[^1] a p-retry wrapper retries any error three times, and the handler
answers every failure with a bare `nak`, which JetStream
[redelivers at once](https://docs.nats.io/learn/jetstream/acknowledgment). JetStream's own default
for `max_deliver` is `-1`, no limit at all, so the harness sets it to four to make the count finite.

`vendor.ts` is a fake of Stripe's customer endpoint that fails on demand and counts every call.
`bus.ts` models a JetStream consumer from its documented rules: an unanswered delivery comes back
after `ack_wait`, `nak` comes back at once, and `term` ends it.

```bash
node calls.ts stacked
```

```text output
every layer retries: stripe-node 2 retries, p-retry 3, JetStream max_deliver 4

failure                           vendor calls  customers created
connection reset on every call    48            0
429 on every call                 16            0
503 on every call                 48            0
worker crashes once after create  2             2
```

The counts multiply. A reset or a 503 costs three attempts inside stripe-node, four calls from
p-retry, and four deliveries from JetStream, which is 48.[^2] A 429 costs 16, because stripe-node
never retries a 429 unless Stripe's `Stripe-Should-Retry` header asks it to. The last row is the
expensive one. Stripe created the customer, the worker died before its `ack`, and the redelivered
message created a second customer under a new key.

## Give each failure class one owner

Give each class to the layer that knows most about it, and switch it off everywhere else.
Network errors and 5xx go to stripe-node, which reuses the first attempt's idempotency key and
obeys `Stripe-Should-Retry`. 429 responses go to the provider, because stripe-node leaves them
alone and Stripe's [rate limit documentation](https://docs.stripe.com/rate-limits) tells you to
retry them with backoff and jitter. Crashes go to JetStream, the only layer that outlives the process.

```ts title="provider.ts"
// The provider owns 429 and nothing else. stripe-node has already retried network errors
// and 5xx inside the call, so a retry here would multiply its attempts.
export function customersProvider(stripe: Stripe, { budget = new RetryBudget(), minTimeout = 500 }: Options = {}) {
  return {
    async create(params: { email: string }, idempotencyKey: string) {
      const customer = await pRetry(() => stripe.customers.create(params, { idempotencyKey }), {
        retries: 3,
        minTimeout,
        randomize: true,
        onFailedAttempt: ({ error }) => { if (isRateLimit(error)) budget.spend() },
        shouldRetry: ({ error }) => isRateLimit(error) && budget.allows(),
      })
      budget.earn()
      return customer
    },
  }
}
```

`shouldRetry` is the switch. It answers yes only for a 429 and only while the budget allows, so a
503 that reaches the provider goes straight back to the handler. The key is passed in rather than
left to stripe-node, which mints a new one for every call it makes, and a p-retry attempt is a new
call.

The budget caps the process, not the call. `RetryBudget` is a token bucket: each 429 spends a
token, each success earns a tenth of one back, and retries stop while half the bucket is
spent.[^3]

```bash
node storm.ts
```

```text output
20 orders while Stripe answers 429 to every call
  per-call cap only            80 vendor calls
  per-call cap and budget      23 vendor calls
```

With the per-call cap alone, every order makes four calls to a vendor that is already refusing
them. With the budget, the first order uses its three retries and the other nineteen give up after
one call each.

## Settle vendor failures before the bus sees them

The handler is where JetStream gets switched off for everything but a crash. It derives the key
from the order, records a vendor failure, and answers it with `term`, which tells the server never
to deliver that message again.

```ts title="handler.ts"
export function onOrder(customers: Customers, store: Store) {
  return async (msg: Msg) => {
    const order = msg.json<Order>()
    // One key per order and operation, never per attempt or per delivery. A redelivered
    // message sends the key the first delivery sent, so Stripe replays the first answer.
    const key = `order-${order.id}-create-customer`
    let customer
    try {
      customer = await customers.create({ email: order.email }, key)
    } catch (err) {
      // stripe-node and the provider have spent their retries. A redelivery repeats them all.
      await store.failed(order.id, (err as Error).message)
      return msg.term('vendor call failed')
    }
    // A crash between the create and the ack leaves the message unanswered, so it comes back.
    await store.save(order.id, customer.id)
    msg.ack()
  }
}
```

A crash sends no answer at all, so it is the one case that still reaches redelivery. The message
returns after `ack_wait` and runs the create again with the same key.

```bash
node calls.ts owners
```

```text output
one owner per class: stripe-node for network and 5xx, p-retry for 429, JetStream for crashes

failure                           vendor calls  customers created
connection reset on every call    3             0
429 on every call                 4             0
503 on every call                 3             0
worker crashes once after create  2             1
```

Every class now costs what its one owner allows: three attempts from stripe-node, four from
p-retry, two deliveries for the crash. The second create in the crash row carried the first one's
key, so the fake replayed `cus_1` instead of creating `cus_2`, as Stripe does when it answers with
`Idempotent-Replayed: true`.

## Check it worked

Run the suite. It asserts the counts for both setups, the budget's 23 calls, and the replay after a
crash. Two more tests pin behavior of the tools themselves: stripe-node retries a closed connection
once even at `maxNetworkRetries: 0`, and a slow handler gets a second delivery unless it calls
`working()`.

```bash
node --test retries.test.ts
```

```text output
1..7
# tests 7
# suites 0
# pass 7
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 1903.093148
```

`# pass 7` with `# fail 0` is the result to expect.

## When it goes wrong

A customer appears twice after a worker restart. The redelivered message ran the create with a key
stripe-node minted for that call, so Stripe saw a new request. Derive the key from the order and the
operation, as Stripe's guidance on [handling errors at the HTTP level](https://docs.stripe.com/error-low-level)
suggests with the id of a shopping cart.

```bash
node pitfall.ts
```

```text output
key minted by stripe-node for each call
  call 1  stripe-node-retry-1036b498-2848-4597-b7d8-1761feaeaedd  created cus_1
  call 2  stripe-node-retry-fec5a937-4e53-44eb-96e8-78d819a7370f  created cus_2
  customers created: 2
key derived from the order
  call 1  order-1001-create-customer  created cus_1
  call 2  order-1001-create-customer  replayed cus_1
  customers created: 1
```

A second worker runs the same message while the first is still retrying. stripe-node's defaults
allow one call three attempts of up to 80 seconds each, over four minutes in all, against a default
`ack_wait` of 30 seconds. Lower stripe-node's `timeout`, raise `ack_wait` past the worst case, or
call `msg.working()` while the handler waits.

Setting `maxNetworkRetries: 0` does not switch stripe-node off. A closed connection is still
retried once, and its README says so in a note. Count two calls per reset when you size a budget.

## When not to do this

Do not add a retry layer to fix a count you have not measured. If stripe-node is the only thing
calling Stripe and nothing redelivers, its retries are the whole story, and a p-retry wrapper adds
a classification to maintain for no gain.

Do not give 429 responses to the provider when the vendor asks for long waits. A wait longer
than `ack_wait` belongs to the bus, where a `nak` with a delay frees the worker while it waits. The
provider must then stop retrying them altogether.

Do not switch JetStream redelivery off. A worker that dies mid-message has no other way back, and
with `term` on every vendor failure, redelivery only ever answers a crash.

Do not copy this split to a vendor whose SDK does not retry. The provider then owns network errors
and 5xx as well, under the same rule: one owner per class, and a key that survives every attempt.
If the vendor accepts no idempotency key, retry its reads only, because a write that timed out may
already have landed.

## Related how-tos

- [Make queue consumer writes safe under redelivery](/howto/queue-consumer-writes-under-at-least-once)

- [Audit SDK retry and timeout defaults before adoption](/howto/audit-sdk-retry-and-timeout-defaults)

- [Enforce one total deadline across all retry attempts](/howto/enforce-a-total-deadline-across-retry-attempts)

- [Migrate a hand-written vendor client to a provider plugin](/howto/migrate-a-vendor-client-to-a-provider)

- [Map vendor error responses onto message errors](/howto/map-vendor-errors-onto-message-errors)

## Last verified

Verified 2026-10-21 against Node 22.22.0, stripe 23.0.0, and p-retry 8.0.1. Every output block is
what the command preceding it printed, run in the page's code directory after `npm ci`. The vendor
is a local fake of Stripe's customer endpoint, so no request reached Stripe. JetStream is modelled
in process from its documented consumer rules, so no NATS server ran. The harness shortens
stripe-node's sleep between retries to keep the run short, which changes when its retries happen
but not how many there are.

[^1]: The stripe-node [README](https://github.com/stripe/stripe-node#network-retries) still gives
the default for `maxNetworkRetries` as 1, and its prose says that as of version 13 the library
makes one reattempt. [Release 17.0.0](https://github.com/stripe/stripe-node/releases/tag/v17.0.0),
published in October 2024, raised the default to 2 and the longest wait between attempts from two
seconds to five, under the heading "Other Breaking changes in the SDK". The code follows the
release notes, so the options table describes the library as it stood from version 13 to 16.

[^2]: The Amazon Builders' Library sets out the same arithmetic in its article on
[timeouts, retries, and backoff with jitter](https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter).
It describes a five-deep stack with three retries at each layer, and puts the load on the database
at 243 times. That figure is three to the fifth power, so it counts three tries per layer. Three
retries would be four tries, and four to the fifth power is 1024. The article's advice holds on
either reading: retry at a single point in the stack.

[^3]: Google's SRE book describes the same idea in its chapter on
[handling overload](https://sre.google/sre-book/handling-overload/). It sets a per-request budget of
up to three attempts, and a per-client budget that allows a retry only while retries stay below 10%
of requests. The bucket in `provider.ts` reaches a similar cap by spending and earning tokens rather
than by counting a ratio, which is the form the Amazon article says the AWS SDK adopted in 2016.