Software Development

Protocol Buffers, Schema Evolution, and the Quiet Discipline IDLs Impose on Distributed Teams

Comparing how protobuf, Avro, and Cap’n Proto solve the same backward-compatibility problem

Nobody deploys a distributed system all at once. The protobuf team says it plainly in their best practices guide: clients and servers are never updated at exactly the same time, even when you try. Consequently, every schema change meets old data and old code sooner or later.

An interface definition language (IDL) exists to manage that meeting. However, the three best-known binary IDLs manage it in three very different ways. This article compares them, then backs the comparison with a small measurement I ran myself.

The Real Problem: Two Versions in One Room

Two directions matter here. First, backward compatibility means new code can read data that old code wrote. Second, forward compatibility means old code can survive data that new code wrote. In practice, a rolling deployment needs both at once.

That is why the interesting question is not syntax. Instead, the question is simple: what makes a field “the same field” across versions? Each format answers it differently, and each answer quietly shapes how your team works.

Three Answers to One Question

Protobuf: The Field Number

In protobuf, a field is its number. The names never travel on the wire, so the language guide is blunt: field numbers should never be reused. Reusing one can cause a parse error at best, and silent data corruption at worst.

Therefore, when you delete a field, you add its number to a reserved list. The compiler then rejects any future attempt to use it. Meanwhile, the same guide notes that numbers 1 to 15 take one byte, whereas 16 to 2047 take two. As a result, teams save the low numbers for frequently set fields.

Forward compatibility comes mostly for free. Since version 3.5, proto3 parsers keep unknown fields and write them back out, as the 3.5 changelog records. Finally, the best practices guide gives one more rule worth memorizing: never add a required field.

Avro: The Name, Plus the Writer’s Schema

Avro takes the opposite route. Its binary encoding carries no field names and no tags, so, according to the specification, readers depend wholly on the schema used for writing. For that reason, the writer’s schema must always travel with the data, either inside a file or by reference.

At read time, Avro compares the writer’s schema with the reader’s schema and resolves the difference. Fields are matched by name, and order does not matter. If the writer has a field the reader lacks, the value is skipped. Conversely, if the reader wants a field the writer never wrote, the reader’s default fills the gap. Without a default, resolution fails.

Renames are handled with aliases, and some type changes are allowed through promotion, for example int to long. You can see the rules in action with this short script, which I tested in a clean virtual environment:

python3 -m venv avro-demo && . avro-demo/bin/activate
pip install fastavro
python3 - <<'EOF'
import io
from fastavro import schemaless_writer, schemaless_reader, parse_schema

def user(extra=None):
    f = [{"name": "id", "type": "long"}, {"name": "name", "type": "string"}]
    return parse_schema({"type": "record", "name": "User", "fields": f + (extra or [])})

v1 = user()
v2 = user([{"name": "country", "type": "string", "default": "unknown"}])
bad = user([{"name": "country", "type": "string"}])

buf = io.BytesIO()
schemaless_writer(buf, v1, {"id": 7, "name": "Ria"})

buf.seek(0)
print(schemaless_reader(buf, v1, v2))   # {'id': 7, 'name': 'Ria', 'country': 'unknown'}
buf.seek(0)
try:
    schemaless_reader(buf, v1, bad)
except Exception as e:
    print(type(e).__name__, "-", e)     # SchemaResolutionError - No default value for field country in User
EOF

Notice what happened. The new field with a default worked, while the one without a default failed loudly. In other words, Avro turns a compatibility rule into a runtime check.

Cap’n Proto: The Ordinal in the File

Cap’n Proto makes the schema file itself the history. According to the schema language guide, the @N numbers must run consecutively from zero, in the order the fields were added. Unlike protobuf, you cannot skip numbers.

The evolution rules follow naturally. New fields need an ordinal larger than all previous ones. Names can change freely as long as the numbers stay put. However, you cannot change a field’s number, type, or default value. Removal is not on the guide’s list of safe changes, and anything unlisted should be assumed unsafe. So, in practice, you simply stop using a field. That last point is my reading of the guide, not a quoted rule.

Side by Side

QuestionProtobufAvroCap’n Proto
What identifies a field?Field numberField name (aliases for renames)Ordinal (@N)
Adding a fieldPick a new numberAdd it with a defaultUse the next, larger ordinal
Removing a fieldDelete it and reserve the numberOld values are ignored by readers that dropped itNot on the safe list, so stop using it
RenamingSafe in binary; breaks JSON and text formatsSafe with an aliasSafe if the ordinal stays
Needs the writer’s schema?NoYes, alwaysNo

What It Costs on the Wire

Compatibility is not free, so I measured the price. I encoded one small record (an id, a name, an email, a flag, and a score) in all three formats. I used Python with the protobuf 7.36.2, fastavro 1.13.1, and pycapnp 2.2.4 packages. Because sizes are properties of the formats, other languages should produce the same byte counts.

Chart 1. Encoded size of the same record, in bytes. Avro excludes the schema, which must travel separately. Cap’n Proto includes its 8-byte segment table and word alignment.

At first glance, Cap’n Proto looks wasteful for a tiny message. That is expected, since it pads everything to 8-byte words so that data can be read in place. Its packed form closes most of the gap.

Next, I added extra integer fields that nobody sets, which mimics a schema that grows over the years. The results below show what each design pays for fields that exist but stay empty.

Chart 2. Encoded size as unused int64 fields are added, in bytes. Measured with 0, 5, 10, 20, and 40 extra fields. One record shape only, so treat it as an illustration, not a verdict.

The pattern is clear. Protobuf stayed at 37 bytes, because proto3 omits default values. Avro grew by one byte per field, since a zero still has to be written. Cap’n Proto grew by 8 bytes per field unpacked, going from 80 to 400 bytes at 40 fields. Packed, however, it moved only from 42 to 44.

Where the Discipline Lives

Here is the quiet part. Each tool puts the burden of compatibility in a different place. With protobuf, the discipline lives in people: someone must remember to reserve the number during code review. With Avro, it lives in the runtime, because resolution fails if a default is missing. With Cap’n Proto, it lives in the file, which only ever grows at the bottom.

Moreover, none of these choices is neutral. Protobuf favors independent teams that rarely share a schema registry. Avro favors pipelines where the writer’s schema always travels with the data. Cap’n Proto favors teams that want to read data in place and accept a stricter file.

Schema evolution checklist

  • Protobuf: never reuse a field number, and reserve numbers you delete.
  • Protobuf: do not add required fields.
  • Avro: give every new field a default.
  • Cap’n Proto: only append new ordinals, and never change a type or default.
  • All three: test old readers against new data, and new readers against old data, before every release.

What We Have Learned

All three formats solve backward compatibility, yet each one decides differently what a field really is: a number in protobuf, a name resolved against the writer’s schema in Avro, and an ordinal in an append-only file in Cap’n Proto. Those choices decide where the discipline lives, whether in code review, at runtime, or in the schema file itself. Our small measurement showed the trade-off in bytes: protobuf and packed Cap’n Proto stayed flat as empty fields piled up, Avro grew by one byte per field, and unpacked Cap’n Proto grew by eight. Therefore, the best choice depends less on speed claims and more on which kind of discipline your team can actually sustain.

Eleftheria Drosopoulou

Eleftheria is an Experienced Business Analyst with a robust background in the computer software industry. Proficient in Computer Software Training, Digital Marketing, HTML Scripting, and Microsoft Office, they bring a wealth of technical skills to the table. Additionally, she has a love for writing articles on various tech subjects, showcasing a talent for translating complex concepts into accessible content.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Back to top button