Schema evolution is the discipline of changing serialized data contracts without breaking producers, consumers, stored records, or rolling deploys. Binary encodings such as Protocol Buffers, Avro, Thrift, and FlatBuffers save bytes by replacing repeated field names with schema-derived structure. That efficiency creates a new operational question: what happens when one service writes version N while another still reads version N-1?
The compatibility problem is not limited to message queues. It appears in RPC APIs, event logs, object storage files, database snapshots, CDC streams, and generated SDKs. A safe schema change must account for both directions of time: new readers consuming old data, old readers consuming new data, and mixed deployments where both happen simultaneously.
DDIA chapter 4 uses schema evolution to connect encoding details to production operations. Protobuf and Thrift store field tags in each record so unknown fields can be skipped. Avro omits field tags and instead resolves a writer schema against a reader schema, which makes schema registries central. The right model changes how you add fields, rename fields, remove fields, assign defaults, and enforce compatibility before deploy.
A new reader is backward compatible when it can read data written by an old writer. This is the common requirement when consumers are deployed before producers. In Avro, adding a field requires a default so the new reader can materialize that field when reading old records that do not contain it.
An old reader is forward compatible when it can read data written by a new writer. This matters when producers deploy first or when old consumers lag behind. Protobuf and Thrift support this by encoding numeric field tags and wire types so old readers can skip unknown fields.
Full compatibility means both directions work against the previous schema. Transitive compatibility means the same guarantee is checked across all historical versions, not only the latest one. Transitive checks are stricter and are often appropriate for long-lived event logs where consumers may replay data written months or years ago.
A schema rule is only useful when tied to deploy order. Backward compatibility lets readers upgrade first. Forward compatibility lets writers upgrade first. Full compatibility lets producers and consumers upgrade independently. Schema registries exist to enforce those contracts before incompatible messages reach durable logs.
Protobuf and Thrift assign stable numeric tags to fields. The binary record carries each field’s tag and enough type information for a parser to skip fields it does not understand. That makes adding optional fields straightforward, but it makes tag reuse dangerous because old data may still contain the old meaning.
// Safe: add a new tag.
message UserV2 {
string id = 1;
string email = 2;
optional string display_name = 3;
}
// Unsafe: tag 2 used to mean email in stored data.
message UserBroken {
string id = 1;
string phone_number = 2;
}
// Safer deletion: reserve the old number and name.
message UserV3 {
string id = 1;
reserved 2;
reserved "email";
optional string display_name = 3;
}Avro encodes field values in schema order without storing field tags beside each value. To decode a record, the reader needs the writer schema. Evolution works by resolving the writer schema against the reader schema by field name and aliases, filling defaults where necessary.
{
"type": "record",
"name": "User",
"fields": [
{"name": "id", "type": "string"},
{"name": "email", "type": ["null", "string"], "default": null},
{"name": "displayName", "type": ["null", "string"], "default": null}
]
}A schema registry stores canonical schemas per subject, assigns IDs or versions, and rejects new schemas that violate configured compatibility. Producers embed or transmit the schema ID with each message. Consumers use that ID to retrieve the writer schema, then decode into their expected reader schema.
def register_schema(subject, candidate, compatibility):
history = registry.load_versions(subject)
checks = history if compatibility.endswith("_TRANSITIVE") else history[-1:]
for old in checks:
if compatibility in ("BACKWARD", "FULL", "BACKWARD_TRANSITIVE", "FULL_TRANSITIVE"):
assert can_read(reader=candidate, writer=old)
if compatibility in ("FORWARD", "FULL", "FORWARD_TRANSITIVE", "FULL_TRANSITIVE"):
assert can_read(reader=old, writer=candidate)
return registry.save(subject, candidate)In a rolling deploy, instances using old and new code run at the same time. A safe producer change must not emit data old consumers cannot tolerate unless consumers were upgraded first. A safe consumer change must still read messages already in queues, logs, or databases.
Kafka topics preserve historical data, so compatibility needs to cover replay as well as the live tail. Confluent’s compatibility modes distinguish previous-version checks from transitive checks. Kafka Streams is especially sensitive because an upgraded application may read both new input records and old changelog state.
Avro object container files can include the writer schema once per file, which makes batch data self-describing. For individual messages in a log or database, storing the full schema per record is too expensive, so a small schema ID plus registry lookup is the standard pattern.
The safest workflow separates a breaking logical change into compatible physical steps. First expand the schema with a nullable/defaulted field. Then dual-write or backfill. Then switch readers. Finally stop writing the old field and reserve or deprecate it after all readers have moved.
// Phase 1: expand.
type UserEventV2 struct {
ID string `json:"id"`
Email *string `json:"email,omitempty"`
PrimaryEmail *string `json:"primary_email,omitempty"`
}
// Phase 2: read both during migration.
func EffectiveEmail(e UserEventV2) string {
if e.PrimaryEmail != nil {
return *e.PrimaryEmail
}
if e.Email != nil {
return *e.Email
}
return ""
}Every schema repository should include golden payloads written by older schemas and tests that decode them with the new code. For forward compatibility, keep a lightweight old-reader test or registry compatibility check in CI so accidental tag reuse or missing defaults fails before deployment.
A schema is a shared API, not an implementation detail. Assign ownership per subject or namespace, require review for incompatible changes, and document semantic meaning outside the field name. Encoding compatibility cannot protect you from changing a field’s business meaning while leaving its type unchanged.
amount_cents to mean dollars while leaving it as an integer passes most schema checks and still breaks consumers. Compatibility policy must include human review for semantics.