Protocol Buffers, Schema Evolution, and the Quiet Discipline IDLs Impose on Distributed Teams
Comparing how protobuf, Avro, and Cap’n Proto solve the same backward-compatibility problem
Nobody deploys a distributed system all at once. The protobuf team says it plainly in their best practices guide: clients and servers are never updated at exactly the same time, even when you try. Consequently, every schema change meets old data and old code sooner or later.
An interface definition language (IDL) exists to manage that meeting. However, the three best-known binary IDLs manage it in three very different ways. This article compares them, then backs the comparison with a small measurement I ran myself.
The Real Problem: Two Versions in One Room
Two directions matter here. First, backward compatibility means new code can read data that old code wrote. Second, forward compatibility means old code can survive data that new code wrote. In practice, a rolling deployment needs both at once.
That is why the interesting question is not syntax. Instead, the question is simple: what makes a field “the same field” across versions? Each format answers it differently, and each answer quietly shapes how your team works.
Three Answers to One Question
Protobuf: The Field Number
In protobuf, a field is its number. The names never travel on the wire, so the language guide is blunt: field numbers should never be reused. Reusing one can cause a parse error at best, and silent data corruption at worst.
Therefore, when you delete a field, you add its number to a reserved list. The compiler then rejects any future attempt to use it. Meanwhile, the same guide notes that numbers 1 to 15 take one byte, whereas 16 to 2047 take two. As a result, teams save the low numbers for frequently set fields.
Forward compatibility comes mostly for free. Since version 3.5, proto3 parsers keep unknown fields and write them back out, as the 3.5 changelog records. Finally, the best practices guide gives one more rule worth memorizing: never add a required field.
Avro: The Name, Plus the Writer’s Schema
Avro takes the opposite route. Its binary encoding carries no field names and no tags, so, according to the specification, readers depend wholly on the schema used for writing. For that reason, the writer’s schema must always travel with the data, either inside a file or by reference.
At read time, Avro compares the writer’s schema with the reader’s schema and resolves the difference. Fields are matched by name, and order does not matter. If the writer has a field the reader lacks, the value is skipped. Conversely, if the reader wants a field the writer never wrote, the reader’s default fills the gap. Without a default, resolution fails.
Renames are handled with aliases, and some type changes are allowed through promotion, for example int to long. You can see the rules in action with this short script, which I tested in a clean virtual environment:
python3 -m venv avro-demo && . avro-demo/bin/activate
pip install fastavro
python3 - <<'EOF'
import io
from fastavro import schemaless_writer, schemaless_reader, parse_schema
def user(extra=None):
f = [{"name": "id", "type": "long"}, {"name": "name", "type": "string"}]
return parse_schema({"type": "record", "name": "User", "fields": f + (extra or [])})
v1 = user()
v2 = user([{"name": "country", "type": "string", "default": "unknown"}])
bad = user([{"name": "country", "type": "string"}])
buf = io.BytesIO()
schemaless_writer(buf, v1, {"id": 7, "name": "Ria"})
buf.seek(0)
print(schemaless_reader(buf, v1, v2)) # {'id': 7, 'name': 'Ria', 'country': 'unknown'}
buf.seek(0)
try:
schemaless_reader(buf, v1, bad)
except Exception as e:
print(type(e).__name__, "-", e) # SchemaResolutionError - No default value for field country in User
EOF
Notice what happened. The new field with a default worked, while the one without a default failed loudly. In other words, Avro turns a compatibility rule into a runtime check.
Cap’n Proto: The Ordinal in the File
Cap’n Proto makes the schema file itself the history. According to the schema language guide, the @N numbers must run consecutively from zero, in the order the fields were added. Unlike protobuf, you cannot skip numbers.
The evolution rules follow naturally. New fields need an ordinal larger than all previous ones. Names can change freely as long as the numbers stay put. However, you cannot change a field’s number, type, or default value. Removal is not on the guide’s list of safe changes, and anything unlisted should be assumed unsafe. So, in practice, you simply stop using a field. That last point is my reading of the guide, not a quoted rule.
Side by Side
| Question | Protobuf | Avro | Cap’n Proto |
|---|---|---|---|
| What identifies a field? | Field number | Field name (aliases for renames) | Ordinal (@N) |
| Adding a field | Pick a new number | Add it with a default | Use the next, larger ordinal |
| Removing a field | Delete it and reserve the number | Old values are ignored by readers that dropped it | Not on the safe list, so stop using it |
| Renaming | Safe in binary; breaks JSON and text formats | Safe with an alias | Safe if the ordinal stays |
| Needs the writer’s schema? | No | Yes, always | No |
What It Costs on the Wire
Compatibility is not free, so I measured the price. I encoded one small record (an id, a name, an email, a flag, and a score) in all three formats. I used Python with the protobuf 7.36.2, fastavro 1.13.1, and pycapnp 2.2.4 packages. Because sizes are properties of the formats, other languages should produce the same byte counts.

At first glance, Cap’n Proto looks wasteful for a tiny message. That is expected, since it pads everything to 8-byte words so that data can be read in place. Its packed form closes most of the gap.
Next, I added extra integer fields that nobody sets, which mimics a schema that grows over the years. The results below show what each design pays for fields that exist but stay empty.

The pattern is clear. Protobuf stayed at 37 bytes, because proto3 omits default values. Avro grew by one byte per field, since a zero still has to be written. Cap’n Proto grew by 8 bytes per field unpacked, going from 80 to 400 bytes at 40 fields. Packed, however, it moved only from 42 to 44.
Where the Discipline Lives
Here is the quiet part. Each tool puts the burden of compatibility in a different place. With protobuf, the discipline lives in people: someone must remember to reserve the number during code review. With Avro, it lives in the runtime, because resolution fails if a default is missing. With Cap’n Proto, it lives in the file, which only ever grows at the bottom.
Moreover, none of these choices is neutral. Protobuf favors independent teams that rarely share a schema registry. Avro favors pipelines where the writer’s schema always travels with the data. Cap’n Proto favors teams that want to read data in place and accept a stricter file.
Schema evolution checklist
- Protobuf: never reuse a field number, and reserve numbers you delete.
- Protobuf: do not add required fields.
- Avro: give every new field a default.
- Cap’n Proto: only append new ordinals, and never change a type or default.
- All three: test old readers against new data, and new readers against old data, before every release.
What We Have Learned
All three formats solve backward compatibility, yet each one decides differently what a field really is: a number in protobuf, a name resolved against the writer’s schema in Avro, and an ordinal in an append-only file in Cap’n Proto. Those choices decide where the discipline lives, whether in code review, at runtime, or in the schema file itself. Our small measurement showed the trade-off in bytes: protobuf and packed Cap’n Proto stayed flat as empty fields piled up, Avro grew by one byte per field, and unpacked Cap’n Proto grew by eight. Therefore, the best choice depends less on speed claims and more on which kind of discipline your team can actually sustain.


