How to replay recorded API traffic in an agent regression suite#

Your agent evaluation suite runs against the live orders API. It fails on Tuesday because the sandbox key expired, on Wednesday because the rate limit tripped, and on Thursday because a test order from Monday still exists. The suite reports a regression each time, though the agent code has not changed in a week.

What you get

You will end up with a committed cassette of one task’s HTTP exchanges, a suite that replays it offline and fails on any uncovered request, and a weekly live run. This is for you if an agent’s tools wrap an HTTP API.

Short answer

Run the task once against the real API with nock’s recorder on, keep only the headers that decide a match, pin the scope to one host, and commit the cassette. In the suite, call nock.disableNetConnect(), load the cassette with nock.define, and listen for nock’s no match event, because a fetch no interaction covers gets a 501 rather than an exception. Schedule one live run so the cassette is compared with the API it stands in for.

You will need

Node 22 or later, Python 3.11 or later for the vcrpy comparison, an agent whose tools wrap one HTTP API, and a key for that API on the day you record. Verified 2026-10-21 against Node 22.22.0, nock 15.0.2, Python 3.13.16, and vcrpy 8.3.0. The sample’s agent is scripted: its policy stands in for the model, so the suite replays the API traffic and nothing else. A stand-in server in api.mjs plays the real API during recording, so the recording itself reproduces offline; against your API, the base URL and the key are the real ones.

Approaches compared

Voxgig maintains sdkgen. It is one of four options here, not the recommendation.

The three recorders differ most where a request does not match. nock answers a fetch with a 501 and emits an event. vcrpy raises CannotOverwriteExistingCassetteException in none mode, and in its default once mode it records when the cassette file is missing. Polly.js records whatever is missing until recordIfMissing is false. The sdkgen test feature records nothing: it is a mock seeded by you, with netsim adding failures to it, so it proves your client’s handling and never the API’s shape.

Record once, against the API you will not call again

The agent’s tools are two calls over fetch, with the key in a header the recording must not keep. The order id comes from the task, so segment escapes it into one path segment of the URL.

// An id from the task or the model becomes exactly one path segment: escaped, and never empty or
// a dot segment, which fetch would resolve to another endpoint, called with the same key.
function segment(id) {
  if (typeof id !== 'string' || id === '' || id === '.' || id === '..') throw new TypeError(`not a usable id: ${JSON.stringify(id)}`)
  return encodeURIComponent(id)
}

export function tools({ baseUrl, apiKey }) {
  const headers = { authorization: `Bearer ${apiKey}`, 'content-type': 'application/json' }
  async function call(method, path, body) {
    const res = await fetch(baseUrl + path, { method, headers, body: body ? JSON.stringify(body) : undefined })
    const json = await res.json()
    if (!res.ok) throw new ApiError(res.status, json)
    return json
  }
  return {
    get_order: async (id) => call('GET', `/v1/orders/${segment(id)}`),
    create_refund: (order_id, amount) => call('POST', '/v1/refunds', { order_id, amount }),
  }
}

nock’s recorder leaves request headers out unless you ask, and when you ask it records authorization with the rest.¹ record.mjs asks, because content-type is worth matching on, and then scrubs. It keeps that one header, drops the response headers that vary between runs, and pins every interaction’s scope to the host the suite will call.

export function scrub(defs) {
  return defs.map((def) => {
    const reqheaders = {}
    let secrets = 0
    for (const [name, value] of Object.entries(def.reqheaders ?? {})) {
      if (name === 'authorization') secrets++
      if (KEEP_REQUEST_HEADERS.includes(name)) reqheaders[name] = value
    }
    const rawHeaders = Object.fromEntries(
      Object.entries(def.rawHeaders ?? {}).filter(([name]) => !DROP_RESPONSE_HEADERS.includes(name)),
    )
    return { ...def, scope: HOST, reqheaders, rawHeaders, secrets }
  })
}

node record.mjs

task outcome: refunded ref_7f3k2m
recorded 2 interactions against https://api.example.com
  GET /v1/orders/ord_1001 -> 200
  POST /v1/refunds {"order_id":"ord_1001","amount":1200} -> 201
authorization headers removed: 2
request headers kept: content-type
wrote cassettes/refund-task.json

The cassette is plain JSON, one object per interaction, and it is what nock.define reads back. The body of the POST is recorded as an object, so the replay matches the refund amount and not only the path.

  {
    "scope": "https://api.example.com",
    "method": "POST",
    "path": "/v1/refunds",
    "body": {
      "order_id": "ord_1001",
      "amount": 1200
    },
    "status": 201,
    "response": {
      "id": "ref_7f3k2m",
      "order_id": "ord_1001",
      "amount": 1200,
      "status": "pending"
    },

Review the file before you commit it, as you would any fixture: a cassette is a copy of what the API said, customer records included.

Replay with the network off, and listen for the miss

nock.disableNetConnect() turns an unmocked host into a thrown NetConnectNotAllowedError. A mocked host is different. When a fetch reaches api.example.com and no interaction matches, nock replies with a 501 whose body carries ERR_NOCK_NO_MATCH, and the agent’s tool sees an API error, not a test failure.² The suite listens for the no match event and asserts on what it collected.

const misses = []
nock.emitter.on('no match', (req) => misses.push(`${req.method} ${req.path ?? req.url}`))

function load() {
  nock.cleanAll()
  misses.length = 0
  if (LIVE) {
    nock.enableNetConnect(new URL(LIVE).host)
    return []
  }
  nock.disableNetConnect()
  return nock.define(JSON.parse(readFileSync(CASSETTE, 'utf8')))
}

The second test asks the agent about an order the cassette never saw, and pins both halves of the behavior: the tool’s 501, and the single miss the event reported.

test('a request the cassette does not cover fails the task instead of leaving the machine', { skip: Boolean(LIVE) }, async () => {
  load()
  await assert.rejects(
    runRefundTask({ baseUrl: base, apiKey: key, orderId: 'ord_2002' }),
    (e) => e instanceof ApiError && e.status === 501 && e.body.code === 'ERR_NOCK_NO_MATCH',
  )
  assert.deepEqual(misses, ['GET https://api.example.com/v1/orders/ord_2002'])
})

The first test asserts misses is empty and every scope isDone(), so a cassette with an interaction the agent stopped making fails too. A recording the code no longer exercises is a second kind of drift.

Run the same task through vcrpy

vcrpy patches http.client, so the standard library’s urllib is enough to compare it on the same three questions.³ vcr_modes.py records against the stand-in with filter_headers, then replays in none mode with another key, with another refund amount, and with a request the cassette does not hold.

cassette = os.path.join(tempfile.mkdtemp(), "refund-task.yaml")
recorder = vcr.VCR(record_mode="once", filter_headers=["authorization"])
with recorder.use_cassette(cassette) as tape:
    order = call("GET", "/v1/orders/ord_1001")
    refund = call("POST", "/v1/refunds", {"order_id": order["id"], "amount": order["amount"]})
    print(f"recorded {len(tape)} interactions, refund {refund['id']}")

PYTHONPATH=vendor python3 vcr_modes.py

recorded 2 interactions, refund ref_7f3k2m
request headers written: connection, content-type, host, user-agent
replay with another key: {'id': 'ord_1001', 'amount': 1200, 'currency': 'EUR'}
POST with another amount: {'id': 'ref_7f3k2m', 'order_id': 'ord_1001', 'amount': 1200}
uncovered request: CannotOverwriteExistingCassetteException: Can't overwrite existing cassette
POST with another amount, body matched: CannotOverwriteExistingCassetteException: Can't overwrite existing cassette

Three answers to read. filter_headers removed the key on the way to disk and left the other request headers in the file. The fourth line is the one to act on. The default match_on compares method, scheme, host, port, path, and query, and the body is not in the list, so a refund for 9999 replayed the recording for 1200. Adding body to match_on turns that into the same exception an uncovered request raises.

Go has the same shape in go-vcr. ModeReplayOnly returns ErrInteractionNotFound for an interaction the cassette lacks, the default matcher takes an option to ignore the Authorization header, and a BeforeSaveHook deletes it before the file is written. Polly.js matches on method, headers, body, and order by default, and excludes headers by name. Its recordIfMissing default is true, so an uncovered request in replay mode goes to the server and is recorded until you set it to false.

Check it worked

Run the suite with the network off. Six tests cover the replay, the uncovered request, the body mismatch, the unmocked host, an order id that tries to leave its segment, and the live run itself. The last starts the stand-in and runs the suite live against both versions of it.

node --test replay.test.mjs

1..6
# tests 6
# suites 0
# pass 6
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 871.670602

# pass 6 with LIVE_API_BASE unset means the cassette alone carried the task. Set LIVE_API_BASE and LIVE_API_KEY, and the first test runs against the real API while the other five skip. Live, it checks the outcome and the steps, not the recorded refund id or nock’s misses: a live API mints a new id, and nock reports every live request as unmatched.

When it goes wrong

The suite stays green after the API changes shape. pitfall.mjs replays the task from the cassette, then runs it live against version 2 of the stand-in, where the order amount moved under a total object as a decimal string.

node pitfall.mjs

replay from the cassette: refunded ref_7f3k2m for 1200
live against the moved API: escalate (order amount missing), steps get_order

The cassette froze the API as it was on the day of recording, and the agent that passes against it escalates every refund in production. The fix is a scheduled live run of the same suite, which is what LIVE_API_BASE is for. live-check.yml runs it weekly under a schedule trigger with the key from a repository secret; copy it under .github/workflows/.

on:
  schedule:
    - cron: '17 6 * * 1'
  workflow_dispatch:

The tool retries the 501. A client with retry on 5xx treats nock’s answer as a transient server error, and each retry is another miss and another 501. The task ends with its retry budget spent and a confusing error. Treat ERR_NOCK_NO_MATCH as fatal in the test client, or let the no match listener fail the test on the first miss.

The cassette carries the key. nock’s recorder wrote it because header recording was on, and a scrub that runs after the fact is the only thing between the recording and the commit. Keep the scrub in record.mjs, and add a CI step that fails when the string authorization appears in a cassette.

When not to do this

Do not record the model’s own replies into the same cassette. The API exchanges change when the API changes; a model’s replies change when the provider changes a snapshot, and a cassette of both goes stale for two unrelated reasons. Script the policy in the regression suite, as here, and evaluate the model’s choices with a separate suite.

Do not let the cassette replace the live run for good. A recording proves the agent still handles what the API said once. The weekly run is what proves the API still says it.

Do not use the sdkgen test feature as a recorder. It is a mock seeded from data you write, and netsim injects failures into it, so together they prove how your client behaves under a 429 or an outage. They know nothing about the real API’s current shape, and a seed is as frozen as a cassette. Pick them when the tools call a generated SDK and the question is your code’s handling, not the wire.

Do not commit a cassette you have not read. It is a copy of what the API returned, customer names and all, and a scrub that removes the key leaves the body as it was.

Related how-tos

• Stub outbound HTTP calls in Node tests with nock

• Run an in-memory mock of your API inside SDK tests

• Mock a generated SDK in your application tests

• Compare agent task success rates across model providers

• Simulate slow and broken networks with Toxiproxy

• Run agent evals in CI without flaky failures

Last verified

Verified 2026-10-21 against Node 22.22.0, nock 15.0.2, Python 3.13.16, and vcrpy 8.3.0 with PyYAML 6.0.3 and wrapt 2.5.1 under vendor/. Every output block is what the command preceding it printed. No request left the machine: the stand-in in api.mjs and the one in vcr_modes.py played the API, and the model was a script. The Polly.js and go-vcr behavior is read from their documentation and not run.