# How to score your API's error messages with a rubric > Sample fifty real error responses from the log, score each body 0 to 2 on six criteria, and publish the score per endpoint before anyone redesigns the errors. Source: https://voxgig.com/howto/score-api-error-messages-with-a-rubric - Audience: api-producer - Level: intermediate - Verified: 2026-10-21 - Published: 2026-10-19 ## Short answer Pull fifty error responses from your logs with a spread across endpoints and status codes. Score each body 0 to 2 on six criteria: field named, fault described, fix given, stable code, documentation link, and the API's dominant shape. The score is a percentage of the points that could apply. Leave a criterion out of the denominator where the status makes it meaningless, report the percentage per endpoint, and re-score after every release. --- ## You will need Node 22 or later, and an export of error responses from your gateway or service logs with the route template, the status, the response headers, and the body of each. Verified 2026-10-21 against Node 22.22.0. The sample runs on a constructed log of 260 error responses across nine routes. The shapes are what a real API accumulates: problem details on two routes, a Stripe-shaped envelope on two, a Google-style `Status` body on one, and bare messages or HTML on the rest. The rubric's wording criteria take their definitions from the [NN/g error-message guidelines](https://www.nngroup.com/articles/error-message-guidelines/), which ask for human-readable language, a precise description of the issue, and constructive advice.[^1] ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [Google AIP-193 error model](https://google.aip.dev/193) | A gRPC or Google-style API where errors already travel as a `Status` with a `details` list | A `details` list to walk, an `ErrorInfo` with a reason in upper snake case and a domain, and canonical codes to map | Your API speaks plain HTTP status codes and a flat JSON body | | [Nielsen Norman Group error-message guidelines](https://www.nngroup.com/articles/error-message-guidelines/) | Scoring the words: whether the message is human-readable, precise, and constructive | Written for interface text, so the machine-readable half of the rubric, codes and links, gets nothing from it | You are scoring shape rather than wording | | [RFC 9457 problem details](https://www.rfc-editor.org/rfc/rfc9457.html) | A standard shape with a registry and a media type, and clients that already parse it | Five members and a type URI you must mint and host, with the fix left to extensions the RFC does not name | The API already has an error envelope every client depends on | | [Stripe error object](https://docs.stripe.com/api/errors) | A flat envelope with `type`, `code`, `message`, `param`, and `doc_url`, which is most of the rubric in five members | A shape owned by one company, with a code list you have to write and maintain yourself | You want a type URI clients can dereference rather than a code list to publish | The three shapes score nearly alike on the rubric and differ in what they cost. Problem details come with a media type and a registry,[^2] and leave the remedy to extensions, so a generic scorer cannot find it. The Google model carries a reason and a Help link inside a `details` list that every client has to walk. Stripe's object is the flattest, and it is one vendor's convention. The NN/g guidelines score none of that. They score the sentence a developer reads. ## Pull fifty responses from the log, not from the docs The docs examples are the errors someone designed. The log holds the errors the API returns, including the ones a gateway or a framework writes on its behalf. Export the error responses with the route template, the status, the headers, and the body, then let `sample.mjs` pick fifty. It groups the log by route and status and takes at most four from each cell in hash order. Every cell keeps its first, and the rest of the fifty places go by hash. The same log gives the same sample on any machine. ```js title="sample.mjs" export function cells(log) { const out = new Map() for (const e of log) { const key = `${e.route} ${e.status}` if (!out.has(key)) out.set(key, []) out.get(key).push(e) } return new Map([...out].sort(([a], [b]) => byCodePoint(a, b))) } /** Up to PER_CELL from every cell. Each cell's first by hash is kept, so a rare status is never * cut, and the places left go to the rest by hash, so a small log still fills TARGET. */ export function sample(log, { perCell = PER_CELL, target = TARGET } = {}) { const firsts = [] const rest = [] for (const [, entries] of cells(log)) { const [first, ...others] = [...entries].sort(byRank).slice(0, perCell) firsts.push(first) rest.push(...others) } const picked = [...firsts.sort(byRank).slice(0, target), ...rest.sort(byRank)].slice(0, target) return picked.sort((a, b) => byCodePoint(a.id, b.id)) } ``` ```bash node sample.mjs ``` ```text output log: 260 error responses, 9 routes, 10 status codes sample: 50, at most 4 per route and status route 400 401 402 403 404 409 422 429 500 502 n GET /v1/customers/{id} . . . . 4 . . . . . 4 GET /v1/invoices/{id} 4 . . . 4 . . . . . 8 GET /v1/reports/{id} . . . 1 . . . 4 . . 5 GET /v1/webhooks . . . . 3 . . . . 1 4 POST /v1/auth/token . 4 . . . . . . . . 4 POST /v1/customers 4 . . . . . . . . . 4 POST /v1/exports 4 . . . . . . . . . 4 POST /v1/invoices . . . . . 2 4 . . . 6 POST /v1/payments 3 . 4 . . . . . 4 . 11 ``` The grid is the first finding. Nine routes answer in ten status codes, and `POST /v1/payments` fails in three of them. `GET /v1/webhooks` returns a 502, which no code of yours wrote. A rare cell keeps a place: `GET /v1/reports/{id}` answered 403 five times in the whole log, and the cut to fifty still keeps one of them. ## Score each body on six criteria, 0 to 2 The rubric is a file, so a second reviewer scores against the same words: ```json title="rubric.json" { "id": "field", "name": "names the offending field or parameter", "applies_to": [400, 404, 409, 422], "levels": { "0": "no field is named", "1": "a field is named in the prose only", "2": "a field is named in a machine-readable member" } }, { "id": "what", "name": "says what was wrong", "levels": { "0": "no message, or only the status phrase", "1": "a message that could be about any request", "2": "a message specific to this occurrence" } }, { "id": "fix", "name": "says how to fix it", "levels": { "0": "no hint", "1": "a hint outside the body, such as Retry-After", "2": "the body says what a correct request looks like" } }, ``` Each criterion is a function in `score.mjs` that reads the parsed body and the headers. The status line decides only which criteria apply: a 401 or a 429 has no offending field, so `field` is left out of its denominator rather than scored 0. That is the trade-off a strict rubric has to make. A bare 401 with a stable code can still reach 100%, and a 400 whose body says only `Bad Request` cannot. ```js title="score.mjs" what(entry, parsed) { const m = messageOf(parsed) if (!m) return 0 const generic = m.toLowerCase() === (REASON[entry.status] ?? '').toLowerCase() || m.split(/\s+/).length < 4 return generic ? 1 : 2 }, fix(entry, parsed) { const m = messageOf(parsed) const help = parsed.kind === 'json' && detail(parsed.body, 'Help') if (help || FIX_WORDS.test(m)) return 2 return entry.headers?.['retry-after'] ? 1 : 0 }, ``` ```bash node report.mjs ``` ```text output sample: 50 responses; dominant shape json:error route n score field what fix code docs shape GET /v1/customers/{id} 4 83% 2.0 2.0 0.0 2.0 2.0 2.0 GET /v1/invoices/{id} 8 75% 1.0 2.0 1.0 2.0 2.0 1.0 GET /v1/reports/{id} 5 24% n/a 1.0 0.8 0.2 0.0 0.4 GET /v1/webhooks 4 0% 0.0 0.0 0.0 0.0 0.0 0.0 POST /v1/auth/token 4 30% n/a 1.0 0.0 1.0 0.0 1.0 POST /v1/customers 4 83% 2.0 2.0 0.0 2.0 2.0 2.0 POST /v1/exports 4 100% 2.0 2.0 2.0 2.0 2.0 2.0 POST /v1/invoices 6 81% 1.3 2.0 1.3 2.0 2.0 1.0 POST /v1/payments 11 19% 0.0 1.0 0.0 0.0 0.0 1.0 all 50 52% lowest: GET /v1/webhooks at 0%, for example 502 "

502 Bad Gateway