How to score your API's error messages with a rubric#

The team agrees the error messages are bad, but cannot say which endpoint is worst or whether last quarter’s fixes helped. The docs show tidy examples. The log shows a 400 whose body says Bad Request, a 404 that arrives as HTML from the gateway, and a 401 that is fine. The redesign is about to start from the examples.

What you get

You will end up with a rubric, a sampler that pulls fifty responses from the log, and a report that scores every endpoint as a percentage beside the three reference shapes. This is for you if you own an API and want a number before the redesign.

Short answer

Pull fifty error responses from your logs with a spread across endpoints and status codes. Score each body 0 to 2 on six criteria: field named, fault described, fix given, stable code, documentation link, and the API’s dominant shape. The score is a percentage of the points that could apply. Leave a criterion out of the denominator where the status makes it meaningless, report the percentage per endpoint, and re-score after every release.

You will need

Node 22 or later, and an export of error responses from your gateway or service logs with the route template, the status, the response headers, and the body of each. Verified 2026-10-21 against Node 22.22.0. The sample runs on a constructed log of 260 error responses across nine routes. The shapes are what a real API accumulates: problem details on two routes, a Stripe-shaped envelope on two, a Google-style Status body on one, and bare messages or HTML on the rest. The rubric’s wording criteria take their definitions from the NN/g error-message guidelines, which ask for human-readable language, a precise description of the issue, and constructive advice.¹

Approaches compared

The three shapes score nearly alike on the rubric and differ in what they cost. Problem details come with a media type and a registry,² and leave the remedy to extensions, so a generic scorer cannot find it. The Google model carries a reason and a Help link inside a details list that every client has to walk. Stripe’s object is the flattest, and it is one vendor’s convention. The NN/g guidelines score none of that. They score the sentence a developer reads.

Pull fifty responses from the log, not from the docs

The docs examples are the errors someone designed. The log holds the errors the API returns, including the ones a gateway or a framework writes on its behalf. Export the error responses with the route template, the status, the headers, and the body, then let sample.mjs pick fifty. It groups the log by route and status and takes at most four from each cell in hash order. Every cell keeps its first, and the rest of the fifty places go by hash. The same log gives the same sample on any machine.

export function cells(log) {
  const out = new Map()
  for (const e of log) {
    const key = `${e.route} ${e.status}`
    if (!out.has(key)) out.set(key, [])
    out.get(key).push(e)
  }
  return new Map([...out].sort(([a], [b]) => byCodePoint(a, b)))
}

/** Up to PER_CELL from every cell. Each cell's first by hash is kept, so a rare status is never
 * cut, and the places left go to the rest by hash, so a small log still fills TARGET. */
export function sample(log, { perCell = PER_CELL, target = TARGET } = {}) {
  const firsts = []
  const rest = []
  for (const [, entries] of cells(log)) {
    const [first, ...others] = [...entries].sort(byRank).slice(0, perCell)
    firsts.push(first)
    rest.push(...others)
  }
  const picked = [...firsts.sort(byRank).slice(0, target), ...rest.sort(byRank)].slice(0, target)
  return picked.sort((a, b) => byCodePoint(a.id, b.id))
}

node sample.mjs

log: 260 error responses, 9 routes, 10 status codes
sample: 50, at most 4 per route and status

route                     400  401  402  403  404  409  422  429  500  502    n
GET /v1/customers/{id}      .    .    .    .    4    .    .    .    .    .    4
GET /v1/invoices/{id}       4    .    .    .    4    .    .    .    .    .    8
GET /v1/reports/{id}        .    .    .    1    .    .    .    4    .    .    5
GET /v1/webhooks            .    .    .    .    3    .    .    .    .    1    4
POST /v1/auth/token         .    4    .    .    .    .    .    .    .    .    4
POST /v1/customers          4    .    .    .    .    .    .    .    .    .    4
POST /v1/exports            4    .    .    .    .    .    .    .    .    .    4
POST /v1/invoices           .    .    .    .    .    2    4    .    .    .    6
POST /v1/payments           3    .    4    .    .    .    .    .    4    .   11

The grid is the first finding. Nine routes answer in ten status codes, and POST /v1/payments fails in three of them. GET /v1/webhooks returns a 502, which no code of yours wrote. A rare cell keeps a place: GET /v1/reports/{id} answered 403 five times in the whole log, and the cut to fifty still keeps one of them.

Score each body on six criteria, 0 to 2

The rubric is a file, so a second reviewer scores against the same words:

    { "id": "field", "name": "names the offending field or parameter",
      "applies_to": [400, 404, 409, 422],
      "levels": { "0": "no field is named", "1": "a field is named in the prose only", "2": "a field is named in a machine-readable member" } },
    { "id": "what", "name": "says what was wrong",
      "levels": { "0": "no message, or only the status phrase", "1": "a message that could be about any request", "2": "a message specific to this occurrence" } },
    { "id": "fix", "name": "says how to fix it",
      "levels": { "0": "no hint", "1": "a hint outside the body, such as Retry-After", "2": "the body says what a correct request looks like" } },

Each criterion is a function in score.mjs that reads the parsed body and the headers. The status line decides only which criteria apply: a 401 or a 429 has no offending field, so field is left out of its denominator rather than scored 0. That is the trade-off a strict rubric has to make. A bare 401 with a stable code can still reach 100%, and a 400 whose body says only Bad Request cannot.

  what(entry, parsed) {
    const m = messageOf(parsed)
    if (!m) return 0
    const generic = m.toLowerCase() === (REASON[entry.status] ?? '').toLowerCase() || m.split(/\s+/).length < 4
    return generic ? 1 : 2
  },
  fix(entry, parsed) {
    const m = messageOf(parsed)
    const help = parsed.kind === 'json' && detail(parsed.body, 'Help')
    if (help || FIX_WORDS.test(m)) return 2
    return entry.headers?.['retry-after'] ? 1 : 0
  },

node report.mjs

sample: 50 responses; dominant shape json:error

route                     n  score   field  what   fix  code  docs shape
GET /v1/customers/{id}    4    83%     2.0   2.0   0.0   2.0   2.0   2.0
GET /v1/invoices/{id}     8    75%     1.0   2.0   1.0   2.0   2.0   1.0
GET /v1/reports/{id}      5    24%     n/a   1.0   0.8   0.2   0.0   0.4
GET /v1/webhooks          4     0%     0.0   0.0   0.0   0.0   0.0   0.0
POST /v1/auth/token       4    30%     n/a   1.0   0.0   1.0   0.0   1.0
POST /v1/customers        4    83%     2.0   2.0   0.0   2.0   2.0   2.0
POST /v1/exports          4   100%     2.0   2.0   2.0   2.0   2.0   2.0
POST /v1/invoices         6    81%     1.3   2.0   1.3   2.0   2.0   1.0
POST /v1/payments        11    19%     0.0   1.0   0.0   0.0   0.0   1.0
all                      50    52%

lowest: GET /v1/webhooks at 0%, for example 502 "<html><body><h1>502 Bad Gateway</h1></body></htm"

reference shapes, one example each, shape left out
  RFC 9457 problem details     6/8
  Google AIP-193 Status        10/10
  Stripe error object          8/10

Read the table by column before by row. fix is the weakest column: five routes score 0 on it, including the two Stripe-shaped routes that pass everything else, because Missing required param: email. names the field but not what a valid one looks like. shape gives the problem-details routes a 1, because this API’s dominant shape is the error envelope, and consistency is measured against your own API rather than against the RFC. POST /v1/payments scores 19% on eleven responses, the most of any route in the sample, which makes it the first endpoint to fix. POST /v1/exports scores 100% with a reason of INVALID_FORMAT and a Help link.³ The example in RFC 9457 scores 6 of 8, because its remedy sits in an extension member that a scorer cannot know is a remedy.⁴

Check it worked

Eleven tests pin the rubric with node:test. Two matter most. The terse 401 keeps a denominator of 10 rather than 12. The status line alone gives every route the same score within two points, while the bodies spread from 0% to 100%. The last three keep a rare cell in the sample, draw the same sample under a Danish locale, and read a JSON string body as a message.

node --test score.test.mjs

1..11
# tests 11
# suites 0
# pass 11
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 380.488389

If # pass 11 drops after an edit to rubric.json, a criterion lost a level or changed its id.

When it goes wrong

Every route scores the same. The scorer read the status line and ignored the body, which is where all six criteria live. A status line carries a code and a reason phrase, and a reason phrase is one point on one criterion:

node pitfall.mjs

scored on the status line only
  GET /v1/customers/{id}     8%
  GET /v1/invoices/{id}      8%
  GET /v1/reports/{id}      10%
  GET /v1/webhooks           9%
  POST /v1/auth/token       10%
  POST /v1/customers         8%
  POST /v1/exports           8%
  POST /v1/invoices          8%
  POST /v1/payments          9%

scored on the body
  GET /v1/customers/{id}    83%
  GET /v1/invoices/{id}     75%
  GET /v1/reports/{id}      24%
  GET /v1/webhooks           0%
  POST /v1/auth/token       30%
  POST /v1/customers        83%
  POST /v1/exports         100%
  POST /v1/invoices         81%
  POST /v1/payments         19%

status line: every route from 8% to 10%; body: from 0% to 100%
criteria read from the status line: 0 of 6

The two points of spread in the first table come from the denominator, because field does not apply to a 401 or a 429. Everything that distinguishes a route is in the second table.

The problem-details routes score lower than the bare-message routes on shape. The dominant shape is whatever most of the sample uses, and on this API that is the error envelope. Read the column as a measure of consistency rather than of merit, and settle which shape wins in the design work that follows, with one schema for every error response.

Two reviewers score the same body differently. The levels in rubric.json are too loose, or one reviewer read the docs example instead of the log entry. Score a shared ten first, compare, and tighten the level wording before anyone scores the other forty.

When not to do this

Do not score the docs examples. They are the errors someone designed, and they score well because they were written to be read. The sample comes from the log or it measures nothing.

Do not redesign from the score. The rubric says which endpoint is worst and on which criterion. The design of the replacement belongs to the error-handling pages, starting with validation errors a client can map to a form.

Do not report the overall 52%. Eleven payment responses at 19% and four export responses at 100% average to a number no endpoint has. Publish the per-route table, and re-score the same fifty cells after each release so that the table has a direction.

Do not add a seventh criterion for tone. Tone is the criterion two reviewers disagree on most, and six criteria are enough to rank endpoints.

Related how-tos

• Verify every error response matches one schema

• Return validation errors a client can map to a form

• Design error responses an autonomous agent can act on

• Choose developer experience metrics for your API

• Run a developer satisfaction survey for your API

Last verified

Verified 2026-10-21 against Node 22.22.0. Every output block is what the command preceding it printed. The log is a constructed fixture of 260 responses, not a real API’s quarter. The RFC 9457 reference body is the example printed in the RFC. The AIP-193 and Stripe reference bodies are built from each shape’s documented members with one illustrative error each. No Google or Stripe endpoint was called.