# How to run an OpenAPI document as a contract test with Specmatic > Keep one JSON example per documented response beside the document, run Specmatic in strict mode against the service, and fail the build when one is missing. Source: https://voxgig.com/howto/openapi-as-a-contract-test-with-specmatic - Audience: api-producer - Level: intermediate - Languages: javascript, java - Verified: 2026-10-21 - Published: 2026-10-19 ## Short answer Put one JSON example per documented response in a directory named `orders_examples` beside `orders.yaml`, start the service, and run `specmatic test --strict --testBaseURL= orders.yaml`. Specmatic sends each example's request and checks the response against the document. Set `successCriteria` to `minCoveragePercentage: 100` with `enforce: true` in `specmatic.yaml`, so a documented response with no example is not tested, lowers coverage, and fails the run. --- ## You will need Node 22 or later for the service and the scripts, a Java runtime for Specmatic, and an OpenAPI 3 document with a service you can start in CI. Specmatic ships as a jar inside the [`specmatic` npm package](https://registry.npmjs.org/specmatic), so `npm ci` fetches it and the `java` on your `PATH` runs it.[^1] Verified 2026-10-21 against Node 22.22.0, OpenJDK 21.0.12.1 and Specmatic 2.55.3. The sample service is `server.mjs`, an in-memory orders API with two operations, started on a free port by every script here. Nothing here makes a network call: `disableTelemetry` in `specmatic.yaml` stops the usage report Specmatic [posts at exit](https://docs.specmatic.io/references/configuration/telemetry) by default. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [Pact provider verification](https://docs.pact.io/getting_started/verifying_pacts) | Consumers you know publish the interactions they rely on | A pact per consumer, fetched from a broker, a URL, or a path, and a handler for every provider state a pact names | No consumer writes pacts, or the document is the contract you want enforced | | [Schemathesis](https://schemathesis.readthedocs.io/en/stable/) | You want generated requests that probe the schema's edges rather than curated ones | A Python toolchain or its Docker image, many generated requests per operation, and a different set on every run unless you pin `--seed` | Every case must be a fixed request a reviewer can read in a diff | | [Specmatic](https://docs.specmatic.io/spec_driven_api_development/contract_testing) | The document is the contract and every case should be a readable example | A JVM at test time, one JSON file per response to keep in step with the service, and a wrapper that masks the exit code | You have no document, or you want the requests nobody thought to write | Schemathesis finds the request nobody wrote down, at the cost of a Python toolchain and a different set of requests on every run unless you pin a seed. Pact verifies only what a consumer asked for, so it says nothing about an operation no consumer uses yet. Specmatic sits between them: each case is a file a reviewer can read, and each file is a thing that goes stale when the service moves. ## Keep one example per documented response Specmatic reads examples from a directory named after the document, `orders_examples` beside `orders.yaml`, and [picks it up by convention](https://docs.specmatic.io/features/external_examples). A directory elsewhere needs `--examples=`. One file holds one request and the response it should get, and its filename is the example's name in every report. ```json title="orders_examples/get_order_404.json" { "http-request": { "method": "GET", "path": "/orders/ord_nope" }, "http-response": { "status": 404, "headers": { "Content-Type": "application/json" }, "body": { "title": "no such order" } } } ``` The four files cover the four documented responses. A valid order produces the 201 and an order with `quantity: 0` the 400; an id the service seeds produces the 200 and an id it never issues the 404. The 400 example breaks the request schema on purpose, and Specmatic's documentation says it [does not enforce contract validity](https://docs.specmatic.io/spec_driven_api_development/contract_testing) for an example meant to produce a 400. Every other file has to agree with the document, which `specmatic examples validate --spec-file orders.yaml` checks without a running service. ## Run the suite in strict mode against a fresh service `contract-test.mjs` starts `server.mjs` on a free port, runs the jar's `test` command against it, and keeps the output lines that carry no timestamp and no absolute path. ```js title="contract-test.mjs" // Runs one Specmatic command and resolves with its exit code and output. A JVM killed by // a signal closes with code null, and process.exitCode = null exits 0, so that counts as 1. export function specmatic(args, { cwd = HERE } = {}) { return new Promise((resolve, reject) => { const child = spawn('java', ['-jar', JAR, ...args], { cwd, stdio: ['ignore', 'pipe', 'pipe'] }) let output = '' child.stdout.on('data', (chunk) => { output += chunk }) child.stderr.on('data', (chunk) => { output += chunk }) child.on('error', reject) child.on('close', (code) => resolve({ code: code ?? 1, output, lines: summarise(output) })) }) } // The contract test: a fresh service, then `specmatic test` pointed at it. The document // and specmatic.yaml are read from cwd, so a caller can run the same suite over a copy. export async function contractTest({ options = ['--strict'], serverEnv = {}, cwd = HERE } = {}) { const server = await startServer(serverEnv) try { return await specmatic(['test', `--testBaseURL=${server.url}`, ...options, 'orders.yaml'], { cwd }) } finally { server.stop() } } ``` ```bash node contract-test.mjs ``` ```text output Scenario: POST /orders -> 201 with the request from the example 'place_order_201' has SUCCEEDED Scenario: POST /orders -> 400 with the request from the example 'place_order_400' has SUCCEEDED Scenario: GET /orders/(orderId:string) -> 200 with the request from the example 'get_order_200' has SUCCEEDED Scenario: GET /orders/(orderId:string) -> 404 with the request from the example 'get_order_404' has SUCCEEDED |----------------------------------------------------------------------------------------------------------------| | SPECMATIC API COVERAGE SUMMARY | |----------------------------------------------------------------------------------------------------------------| | coverage | path | method | requestContentType | response | responseContentType | remarks | result | |----------|-------------------|--------|--------------------|----------|---------------------|---------|--------| | 100% | /orders | POST | application/json | 201 | application/json | covered | 1p | | | | | | 400 | application/json | covered | 1p | | 100% | /orders/{orderId} | GET | NA | 200 | application/json | covered | 1p | | | | | | 404 | application/json | covered | 1p | |----------------------------------------------------------------------------------------------------------------| | * = Operation not eligible for coverage | | ! = Operation excluded from the run due to a filter | | p = passed tests | | f = failed tests | | w = wip tests | | | | 100% API Coverage reported from 4 operations eligible for coverage | | 100% Absolute Coverage (includes excluded operations that were not tested) | |----------------------------------------------------------------------------------------------------------------| Tests run: 4, Successes: 4, Failures: 0, WIP: 0, Errors: 0 ``` Each scenario line names the operation, the status, and the example file the request came from. The `--strict` flag is what makes the list complete. Without it, Specmatic [generates a positive test](https://docs.specmatic.io/spec_driven_api_development/contract_testing) for an operation with no example, filling the request from the schema, so the gap is covered by a request nobody reviewed. The table is the coverage report. A row reads `covered` when a response matched it, `not tested` when no test ran against it, and `not implemented` when a test ran and the service did not answer as the document says. The harness runs the jar through `java -jar` rather than `npx specmatic`, for a reason shown below. ## Fail the build when a response has no example Coverage is the lever. Under `--strict` an operation with no example is skipped, so it is not covered, and the [report configuration](https://docs.specmatic.io/references/configuration/reports) turns coverage below a threshold into a failed run. ```yaml title="specmatic.yaml" version: 3 specmatic: settings: general: # Specmatic posts usage analytics at exit unless told not to. disableTelemetry: true governance: report: formats: - html outputDirectory: build/reports/specmatic # Under --strict an operation with no example is not tested, so it is not # covered, so a run that skips one fails this threshold. successCriteria: minCoveragePercentage: 100 maxMissedOperationsInSpec: 0 enforce: true ``` `missing-example.mjs` copies the document, this configuration, and three of the four examples into a temporary directory, then runs the same suite from there. ```bash node missing-example.mjs ``` ```text output Scenario: POST /orders -> 201 with the request from the example 'place_order_201' has SUCCEEDED Skipping Scenario: POST /orders -> 400 (requestContentType application/json, responseContentType application/json) - T00004: Generative Disabled Summary: This operation was skipped because it requires schema resiliency to be enabled - T00003: Examples Required in Strict Mode Summary: Strict mode requires at least one example, but none were found for this operation Scenario: GET /orders/(orderId:string) -> 200 with the request from the example 'get_order_200' has SUCCEEDED Scenario: GET /orders/(orderId:string) -> 404 with the request from the example 'get_order_404' has SUCCEEDED |-------------------------------------------------------------------------------------------------------------------| | SPECMATIC API COVERAGE SUMMARY | |-------------------------------------------------------------------------------------------------------------------| | coverage | path | method | requestContentType | response | responseContentType | remarks | result | |----------|-------------------|--------|--------------------|----------|---------------------|------------|--------| | 50% | /orders | POST | application/json | 201 | application/json | covered | 1p | | | | | | 400 | application/json | not tested | | | 100% | /orders/{orderId} | GET | NA | 200 | application/json | covered | 1p | | | | | | 404 | application/json | covered | 1p | |-------------------------------------------------------------------------------------------------------------------| | * = Operation not eligible for coverage | | ! = Operation excluded from the run due to a filter | | p = passed tests | | f = failed tests | | w = wip tests | | | | 75% API Coverage reported from 4 operations eligible for coverage | | 75% Absolute Coverage (includes excluded operations that were not tested) | |-------------------------------------------------------------------------------------------------------------------| Failed the following API Coverage Report success criteria: Total API coverage: 75% is less than the specified minimum threshold of 100%. One or more API Coverage report's success criteria were not met Tests run: 3, Successes: 3, Failures: 0, WIP: 0, Errors: 0 exit code 1 ``` Three tests passed and the run exited 1, which is the behavior a gate needs: the 400 response was never exercised, so its row reads `not tested` and coverage is 75%. The two codes under the skipped scenario come from Specmatic's [rule catalog](https://docs.specmatic.io/rules).[^2] `T00003` is the strict-mode skip. `T00004` says the negative tests Specmatic could generate for a 4xx are off, since they need its schema resiliency mode. An example is therefore the one way a 4xx row gets covered here. ## Read the report when an example goes stale The example for the 200 names `ord_1001`, an order the service seeds at startup. `drift.mjs` starts the service without it and runs the same strict suite. ```bash node drift.mjs ``` ```text output Scenario: POST /orders -> 201 with the request from the example 'place_order_201' has SUCCEEDED Scenario: POST /orders -> 400 with the request from the example 'place_order_400' has SUCCEEDED Scenario: GET /orders/(orderId:string) -> 200 with the request from the example 'get_order_200' has FAILED >> RESPONSE.STATUS R0002: HTTP status mismatch Summary: The HTTP status code does not match the expected status code defined in the specification Specification expected status 200 but response contained status 404 Scenario: GET /orders/(orderId:string) -> 404 with the request from the example 'get_order_404' has SUCCEEDED ``` The run goes on to the table, where the 200 row reads `not implemented` with a result of `1f`, and ends with `Tests run: 4, Successes: 3, Failures: 1, WIP: 0, Errors: 0` and exit code 1. The scenario line names the operation and the example file, and `R0002` names the mismatch, so the report answers which operation and which example without a log. Specmatic also writes an HTML report under `build/reports/specmatic` on every run.[^3] The fix is never to loosen the example. Either the service seeds the record the example names, or a setup step creates it before the run, which is what [Dredd hooks](/howto/stateful-api-tests-with-dredd-hooks) do for the same kind of suite. Review the example files in the same pull request as the document, where a renamed field or a retired fixture id is a visible diff before it is an `R0002`. ## Check it worked `contract.test.mjs` runs the strict suite and asserts the exit code, the totals line, and the coverage line, then runs `examples validate` and asserts that all four files are valid. A third asserts that the harness exits 1 when `java` dies of `SIGKILL`. ```js title="contract.test.mjs" test('every documented response has an example, and every example passes against the service', async () => { const { code, lines } = await contractTest() assert.equal(code, 0, lines.join('\n')) assert.ok(lines.includes('Tests run: 4, Successes: 4, Failures: 0, WIP: 0, Errors: 0'), lines.join('\n')) assert.ok(lines.some((l) => l.includes('100% API Coverage reported from 4 operations')), lines.join('\n')) assert.ok(!lines.some((l) => /T00003/.test(l)), 'no operation was skipped for want of an example') }) ``` ```bash node --test contract.test.mjs ``` ```text output 1..3 # tests 3 # suites 0 # pass 3 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 8139.442258 ``` All three tests pass. The second checks the example files against the document with no service running, so an example that breaks the document fails without a server. `Tests run: 4, Successes: 4` is the literal to look for. ## When it goes wrong The build stays green after a failed contract. `pitfall.mjs` runs the same failing suite as `drift.mjs` through `npx specmatic`, the bin the npm package installs, instead of the jar. ```bash node pitfall.mjs ``` ```text output Scenario: GET /orders/(orderId:string) -> 200 with the request from the example 'get_order_200' has FAILED Tests run: 4, Successes: 3, Failures: 1, WIP: 0, Errors: 0 npx specmatic exit code 0 ``` The jar exited 1 on the same run. The wrapper receives that code and drops it on the way out,[^4] so a CI step built on `npx specmatic test` passes whatever the contract says. Run `java -jar node_modules/specmatic/specmatic.jar test ...`, as `contract-test.mjs` does, or parse the `Tests run:` line yourself. A 4xx row reads `not tested` without `--strict` too. Without the flag Specmatic generates a positive test for an operation with no example and skips the negative ones with `T00002`, Examples Required. Add an example per error response: the request may break the schema, and the response must match it. The run passes with a wrong value in the response. Specmatic validates the response against the document's schema, not against the body written in the example. With the example's `quantity` changed to 9 while the service kept answering 2, the scenario still passed. Contract tests prove shape, so a value that matters needs a test that asserts it. ## When not to do this Do not use Specmatic to police an API you do not own. Its examples describe the provider's behavior from the provider's side, and the service has to be yours to start in CI. For a vendor API, the scheduled runs in [contract test an API you do not own](/howto/contract-test-an-api-you-do-not-own) fit better. Do not read a green strict run as a statement about values. The suite proves that each documented response exists, carries the documented status, and matches the schema. A field with the right type and the wrong number passes. Do not let the examples become a second copy of your fixtures. Each one names records the service must hold, so a suite that needs `ord_1001` to exist has a setup problem the examples cannot solve. Seed the service for the run. Do not pick Specmatic for the cases nobody thought of. Its generated requests are random values that fit the schema, and its negative tests need the resiliency mode. Schemathesis is built for that search and prints a `curl` command that reproduces each failure. ## Related how-tos - [Choose consumer-driven or spec-driven contract tests](/howto/choose-consumer-driven-or-spec-driven-contract-tests) - [Test stateful endpoints with Dredd hooks](/howto/stateful-api-tests-with-dredd-hooks) - [Validate responses against the OpenAPI document at runtime](/howto/validate-responses-against-the-openapi-document) - [Contract test an API you do not own](/howto/contract-test-an-api-you-do-not-own) ## Last verified Verified 2026-10-21 against Node 22.22.0, OpenJDK 21.0.12.1 and Specmatic 2.55.3. Every output block is what the command preceding it printed, after `contract-test.mjs` dropped the lines that carry a timestamp, an absolute path, or a report-writing notice. The observation about response values in the section on what goes wrong comes from a run whose output is not shown, with the example's `quantity` edited. Schemathesis and Pact were read from their documentation, not run. [^1]: The npm registry lists the package's unpacked size as [85,710,486 bytes](https://registry.npmjs.org/specmatic/2.55.3) across 64 files, and the README calls the module a thin wrapper over the Specmatic executable jar. The jar ships inside the same tarball, which is where the 85 megabytes go, so `npm ci` fetches a Java program and leaves finding a Java runtime to you. [^2]: Specmatic numbers its reasons. The [rule catalog](https://docs.specmatic.io/rules) files `T00002` as `Examples Required`, `T00003` as `Examples Required in Strict Mode`, and `T00004` as `Generative Disabled`, with an entry for each under the anchor the console prints beside it. `R0002`, `HTTP status mismatch`, sits in a second series for what the response did. A failure therefore arrives with its citation attached. [^3]: Near the end of every run the console prints `Generating CTRF report using coverage report specifications...`, and the [reports reference](https://docs.specmatic.io/references/configuration/reports) files the [Common Test Report Format](https://ctrf.io/) under the commercial edition. The run here, on the open source license, wrote the HTML report at `build/reports/specmatic/test/html/index.html`, the only file it wrote even with `ctrf` added to `formats`. The CTRF file the console announces belongs to the paid edition, so the announcement is the part of it that ships. [^4]: The command handler in the wrapper's `bin`, in the [`specmatic-node`](https://github.com/specmatic/specmatic-node) repository, ends with `process.exitCode = err.code`. The error it receives is built as `new Error('Command exited with non zero code: ' + code)`, which carries the number in its message and has no `code` property. The assignment therefore stores `undefined`, and Node exits 0. The number leaves the JVM and stops in the wrapper.