How to evolve an Avro event schema without breaking consumers#

You add a field to the order event, deploy the producer, and the fulfillment consumer starts throwing on every record. Its unit tests passed, because they encoded and decoded with the same schema. Nothing compared the changed schema with the one the consumer was built against, and the broker does not care what the bytes mean. The topic holds three days of records it cannot read.

What you get

You will end up with a BACKWARD subject, a producer that adds a defaulted field, a consumer on the previous schema still reading, and proof that the registry refuses the breaking change. This is for you if other teams consume your topic.

Short answer

Register the topic’s Avro schema in a schema registry under a BACKWARD-compatible subject, then add fields only with a default, so a consumer holding the previous schema still resolves every record. Each record carries a zero byte, the schema id, and the Avro bytes; the consumer fetches the writer schema by id and resolves it against its own. The registry refuses a field without a default at registration, which is where the break should surface.

You will need

Node 22 or later, a Kafka topic whose records are Avro, and a schema registry that speaks the Confluent REST API. Verified 2026-10-21 against Node 22.22.0, @kafkajs/confluent-schema-registry 4.1.0, and avsc 5.7.9, installed from npm in the sample directory. The sample runs no broker and no real registry. An in-process stand-in answers the registry routes the client calls, with the compatibility check done by Avro’s resolution rules as avsc implements them. The record values are the buffers a producer would send.

Approaches compared

The registry buys one thing: a change that would break a reader is refused before a producer ships it. The Confluent registry is the one most clients already speak, at the cost of a service and a subject layout you choose once. Apicurio takes more formats behind a compatible API, for a second service to run. AsyncAPI in CI adds no runtime dependency, and for the same reason it never sees the bytes. A version field costs nothing at all and moves the whole burden into every consumer.

Name the subject and register the first schema

A subject is the name the registry keeps versions under, and the compatibility check runs within a subject. The Confluent serializers derive it from the topic, the record, or both, by a subject name strategy. The client here takes the subject as a string, so the sample makes the choice in one module.

export const strategies = {
  // One schema per topic. The default, and right until a topic carries two event types.
  TopicNameStrategy: (topic, schema, { isKey = false } = {}) => `${topic}-${isKey ? 'key' : 'value'}`,
  // One schema per record type, whatever topic it travels on.
  RecordNameStrategy: (topic, schema) => fullName(schema),
  // One schema per record type per topic.
  TopicRecordNameStrategy: (topic, schema) => `${topic}-${fullName(schema)}`,
}

The producer registers the schema once and gets an id. encode then writes the record value in the registry’s wire format: a magic byte of zero, the schema id as a four-byte big-endian integer, and the Avro binary.¹

const subject = strategies.TopicNameStrategy(topic, orderPlacedV1)
const { id } = await registry.register({ type: SchemaType.AVRO, schema: JSON.stringify(orderPlacedV1) }, { subject })
console.log(`\nregistered shop.orders.OrderPlaced under ${subject}: id ${id}, compatibility ${local.levelOf(subject)}`)

// encode() writes the wire format: one zero byte, the schema id as a big-endian int32, then the
// Avro binary. This buffer is what goes in the Kafka record's value.
const value = await registry.encode(id, { orderId: 'ord_1', customerId: 'cus_9', totalCents: 4200 })

node register.mjs

TopicNameStrategy        orders-value
RecordNameStrategy       shop.orders.OrderPlaced
TopicRecordNameStrategy  orders-shop.orders.OrderPlaced

registered shop.orders.OrderPlaced under orders-value: id 1, compatibility BACKWARD
record value: 19 bytes
  byte 0      magic byte  0
  bytes 1-4   schema id   1  (0 0 0 1)
  bytes 5-18  avro        0a6f72645f310a6375735f39d041

decoded by a fresh consumer: {"orderId":"ord_1","customerId":"cus_9","totalCents":4200}
registry requests:
  GET /config/orders-value
  POST /subjects/orders-value/versions
  PUT /config/orders-value
  GET /schemas/ids/1

Fourteen of the nineteen bytes are the order. The other five are what lets a consumer that has never seen this schema read it: it takes the id from the prefix, fetches the writer schema with GET /schemas/ids/1 once, and caches it. The PUT /config is the client setting its default level, BACKWARD, which is also the registry’s default, on a subject that had none.

Pick the strategy before the first producer ships, because changing a subject later means re-registering history. Topic name is right while a topic carries one event type; several types need a record-based strategy, as the last section shows.

Add the field under BACKWARD

BACKWARD means that a consumer using the new schema can read data produced with the last schema, in the words of the Confluent documentation. The check is Avro’s own schema resolution. Fields are matched by name, and a writer’s field the reader lacks is skipped. A reader’s field the writer lacks is filled from the reader’s default or, with no default, is an error. So a field arrives with a default, and a consumer on the previous schema keeps resolving every record.

// The change: a field with a default, registered under BACKWARD. The registry checks that the
// new schema can read what v1 wrote, and only then hands out an id.
const { id: id2 } = await producer.register(asAvro(orderPlacedV2), { subject, compatibility: 'BACKWARD' })
const newBytes = await producer.encode(id2, { orderId: 'ord_2', customerId: 'cus_9', totalCents: 1500, currency: 'USD' })
console.log(`v1 is id ${id1}, v2 is id ${id2}, subject ${subject} is ${local.levelOf(subject)}`)

// The consumer that has not been redeployed still holds v1 as its reader schema. Resolution
// matches fields by name, so the field it does not know is skipped.
const oldConsumer = new SchemaRegistry({ host: local.url })
const readByOld = await oldConsumer.decode(newBytes, { [SchemaType.AVRO]: { readerSchema: orderPlacedV1 } })
console.log(`old consumer, reader v1, new bytes: ${JSON.stringify(readByOld)}`)

// The redeployed consumer holds v2 and rewinds to the start of the topic. The field v1 never
// wrote is filled from the default in its own schema.
const newConsumer = new SchemaRegistry({ host: local.url })
const readByNew = await newConsumer.decode(oldBytes, { [SchemaType.AVRO]: { readerSchema: orderPlacedV2 } })
console.log(`new consumer, reader v2, old bytes: ${JSON.stringify(readByNew)}`)

node evolve.mjs

v1 is id 1, v2 is id 2, subject orders-value is BACKWARD
old consumer, reader v1, new bytes: {"orderId":"ord_2","customerId":"cus_9","totalCents":1500}
new consumer, reader v2, old bytes: {"orderId":"ord_1","customerId":"cus_9","totalCents":4200,"currency":"EUR"}

register v3 (channel, no default): HTTP 409
  Confluent_Schema_Registry - Schema being registered is incompatible with an earlier schema for subject "orders-value", details: no matching field for default-less shop.orders.OrderPlaced.channel
versions of orders-value: [{"id":1,"version":1},{"id":2,"version":2}]

The consumer on v1 reads a v2 record and never sees currency. The consumer on v2 reads a v1 record and gets EUR, the default, for a field the producer never wrote.² The reader schema is passed to decode, and the writer schema comes from the registry by id, which is the whole mechanism. Then v3 adds channel with no default, and the registration is answered with 409, the status the REST API documents for an incompatible schema, before any producer can use it. The subject stays at two versions.

Each format has its own rules. Avro’s are the resolution rules quoted earlier, which is why the check is a short list. The Confluent compatibility guide calls the Protobuf and JSON Schema rules more nuanced, and recommends BACKWARD_TRANSITIVE for Protobuf because adding a message type is not forward compatible.

The same two calls exist in the other clients. In confluent-kafka-python the AvroDeserializer takes the reader schema as schema_str, and in Go the franz-go sr package pairs a Serde with a ConfluentHeader that produces the same five-byte prefix.

Check it worked

The test that matters registers v2 and then v3 under BACKWARD, and asserts that the second attempt is refused with a 409 that names the field.

test('under BACKWARD a field with a default registers and a field without one is refused', async () => {
  const registry = new SchemaRegistry({ host: local.url })
  const { id: id1 } = await registry.register(asAvro(orderPlacedV1), { subject: 'backward-value' })
  const { id: id2 } = await registry.register(asAvro(orderPlacedV2), { subject: 'backward-value', compatibility: 'BACKWARD' })
  assert.notEqual(id1, id2)
  await assert.rejects(registry.register(asAvro(orderPlacedV3NoDefault), { subject: 'backward-value' }), (err) => {
    assert.equal(err.status, 409)
    assert.match(err.message, /incompatible with an earlier schema/)
    assert.match(err.message, /default-less shop\.orders\.OrderPlaced\.channel/)
    return true
  })
  assert.deepEqual(local.versions('backward-value').map((v) => v.version), [1, 2])
})

node --test evolve.test.mjs

1..7
# tests 7
# suites 0
# pass 7
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 304.828737

Seven tests pass. The others pin the five-byte prefix, both reader schemas, and the subject names each strategy produces. One pins the difference between the levels: removing a field passes BACKWARD and fails FORWARD and FULL, since a consumer on the previous schema could not fill it. Another pins the transitive case.

When it goes wrong

The unit tests are green and the registry says no. A test that encodes and decodes with one schema proves that schema agrees with itself, and the registry compares it with what is already registered. The compatibility endpoint answers the same question without registering anything.

node pitfall.mjs

v3 round trip in a unit test:  {"orderId":"ord_3","customerId":"cus_9","totalCents":10,"currency":"EUR","channel":"web"}
v3 against orders-value:       {"is_compatible":false,"messages":["no matching field for default-less shop.orders.OrderPlaced.channel"]}

avsc fromBuffer on the framed value: throws "trailing data"
avsc fromBuffer after 5 bytes:      {"orderId":"ord_1","customerId":"cus_9","totalCents":4200}

OrderCancelled under orders-value: HTTP 409
  Confluent_Schema_Registry - Schema being registered is incompatible with an earlier schema for subject "orders-value", details: no alias found for shop.orders.OrderPlaced
OrderCancelled under shop.orders.OrderCancelled: id 2

Run the compatibility check in CI with verbose=true, so a pull request that adds a field without a default fails there rather than in a deploy.

A plain Avro decoder throws on every record. It was handed the framed value, so it read the magic byte as the first field. Here the result is trailing data; another schema could yield wrong values and no error. Strip the five bytes, or use a registry-aware deserializer, which reads the id and fetches the schema.

A second event type on the topic is refused as incompatible. Under the topic name strategy, both records share the subject orders-value, and OrderCancelled cannot read what OrderPlaced wrote. The record name strategy gives each type its own subject, and the registration goes through.

When not to do this

Do not add a registry to a topic with one producer and one consumer in one repository. A shared schema file and the same compatibility check run in that repository’s tests gives the same protection without a service to run.

Do not set a subject to NONE to get a deploy out. The check that refused the change is the one thing the registry does, and a consumer that cannot read the topic is the outage you bought the registry to prevent.

Do not choose FORWARD because the producer team wants to remove fields. BACKWARD is the Confluent default so that a consumer can be rewound to the start of the topic, and FORWARD gives up that guarantee.³

Do not read BACKWARD as a check against the whole history. It compares the candidate with the latest version only, so a version that drops a default the previous version added passes, though older records become unreadable. BACKWARD_TRANSITIVE checks every earlier version, at the cost of refusing changes the registry would otherwise allow.⁴

Related how-tos

• Describe a Kafka topic with AsyncAPI

• Make queue consumer writes safe under redelivery

• Define a gRPC service in a proto3 file

• Evolve an error contract without breaking clients

Last verified

Verified 2026-10-21 against Node 22.22.0, @kafkajs/confluent-schema-registry 4.1.0, and avsc 5.7.9. Every output block is what the command preceding it printed. The registry in the sample is an in-process stand-in for the routes the client calls, with compatibility judged by the schema resolution in avsc. Its 409 is the status the API reference documents for an incompatible schema, with a message of the stand-in’s own. Confluent Schema Registry, Apicurio Registry, and the Glue registry were not run; their behavior is taken from their documentation. No broker was involved, so the record values are the buffers a producer would send.