Protocol Buffers Wire Encoding

Overview

Protocol Buffers (protobuf) is Google’s language-neutral, platform-neutral serialization format used by gRPC, Google Cloud APIs, and countless internal systems. Unlike JSON, which is text-based and self-describing, protobuf encodes data as a compact binary stream where field names are replaced by numeric tags and values use variable-length encoding. A message that takes 200 bytes in JSON might take 40 bytes in protobuf, with 10x faster serialization and deserialization.

As a concrete example, consider implementing protobuf wire encoding natively in Zig for a gRPC transport layer, bypassing the standard protoc-generated code in favor of hand-written encoders and decoders optimized for specific message types. This approach eliminates the protoc dependency, reduces binary size, and allows the Zig compiler to inline encoding/decoding logic for zero-overhead serialization. Understanding the wire format is essential for anyone implementing custom protobuf codecs, debugging binary protocol issues, or optimizing serialization-heavy workloads.

This document covers the protobuf wire format from the ground up: varint encoding, zigzag encoding for signed integers, wire types, field tags, nested message encoding, and packed repeated fields, with Zig code examples demonstrating custom codec patterns.

Core Concepts

How It Works

Varint Encoding in Zig

The varint encoder writes 7 bits at a time, setting the MSB continuation flag on all bytes except the last:

const ProtoWriter = struct {
    buf: []u8,
    pos: usize = 0,

    pub fn writeVarint(self: *ProtoWriter, value: u64) void {
        var v = value;
        while (v > 0x7F) {
            self.buf[self.pos] = @truncate((v & 0x7F) | 0x80);
            self.pos += 1;
            v >>= 7;
        }
        self.buf[self.pos] = @truncate(v);
        self.pos += 1;
    }

    pub fn writeSignedVarint(self: *ProtoWriter, value: i64) void {
        // ZigZag encode: (n << 1) ^ (n >> 63)
        const zigzag: u64 = @bitCast((value << 1) ^ (value >> 63));
        self.writeVarint(zigzag);
    }
};

const ProtoReader = struct {
    buf: []const u8,
    pos: usize = 0,

    pub fn readVarint(self: *ProtoReader) !u64 {
        var result: u64 = 0;
        var shift: u6 = 0;
        while (true) {
            if (self.pos >= self.buf.len) return error.UnexpectedEof;
            const byte = self.buf[self.pos];
            self.pos += 1;
            result |= @as(u64, byte & 0x7F) << shift;
            if (byte & 0x80 == 0) break;
            shift += 7;
            if (shift >= 64) return error.VarintOverflow;
        }
        return result;
    }

    pub fn readSignedVarint(self: *ProtoReader) !i64 {
        const zigzag = try self.readVarint();
        // Decode ZigZag: (n >>> 1) ^ -(n & 1)
        return @bitCast((zigzag >> 1) ^ (-%( zigzag & 1)));
    }
};

Field Tag Encoding and Decoding

The field tag packs the field number and wire type. The encoder writes the tag before each field value, and the decoder extracts both from the tag varint:

const WireType = enum(u3) {
    varint = 0,
    fixed64 = 1,
    length_delimited = 2,
    start_group = 3,  // deprecated
    end_group = 4,    // deprecated
    fixed32 = 5,
};

const FieldTag = struct {
    field_number: u29,
    wire_type: WireType,

    pub fn encode(self: FieldTag) u32 {
        return (@as(u32, self.field_number) << 3) | @intFromEnum(self.wire_type);
    }

    pub fn decode(tag: u64) !FieldTag {
        return .{
            .field_number = @truncate(tag >> 3),
            .wire_type = @enumFromInt(@as(u3, @truncate(tag & 0x07))),
        };
    }
};

// Writing a field: tag + value
fn writeUint64Field(writer: *ProtoWriter, field_number: u29, value: u64) void {
    const tag = FieldTag{ .field_number = field_number, .wire_type = .varint };
    writer.writeVarint(tag.encode());
    writer.writeVarint(value);
}

fn writeStringField(writer: *ProtoWriter, field_number: u29, value: []const u8) void {
    const tag = FieldTag{ .field_number = field_number, .wire_type = .length_delimited };
    writer.writeVarint(tag.encode());
    writer.writeVarint(value.len);
    @memcpy(writer.buf[writer.pos..][0..value.len], value);
    writer.pos += value.len;
}

Nested Message Encoding

Nested messages are length-delimited: the encoder first serializes the inner message to a temporary buffer to determine its length, then writes the tag, length, and the serialized bytes. An optimization pre-computes the length without actually serializing:

const VectorPoint = struct {
    id: u64,        // field 1, varint
    vector: []f32,  // field 2, packed repeated float
    name: []const u8, // field 3, string

    pub fn encode(self: VectorPoint, writer: *ProtoWriter) void {
        // Field 1: id (varint)
        writeUint64Field(writer, 1, self.id);

        // Field 2: vector (packed repeated float = length-delimited)
        const tag2 = FieldTag{ .field_number = 2, .wire_type = .length_delimited };
        writer.writeVarint(tag2.encode());
        writer.writeVarint(self.vector.len * 4); // 4 bytes per f32
        for (self.vector) |val| {
            const bytes: [4]u8 = @bitCast(val);
            @memcpy(writer.buf[writer.pos..][0..4], &bytes);
            writer.pos += 4;
        }

        // Field 3: name (string)
        if (self.name.len > 0) {
            writeStringField(writer, 3, self.name);
        }
    }

    pub fn decode(reader: *ProtoReader, end: usize) !VectorPoint {
        var point = VectorPoint{ .id = 0, .vector = &.{}, .name = "" };
        while (reader.pos < end) {
            const raw_tag = try reader.readVarint();
            const tag = try FieldTag.decode(raw_tag);
            switch (tag.field_number) {
                1 => point.id = try reader.readVarint(),
                2 => {
                    const len = try reader.readVarint();
                    const float_count = len / 4;
                    // parse float_count f32 values from reader
                    point.vector = try reader.readPackedFloats(float_count);
                },
                3 => {
                    const len = try reader.readVarint();
                    point.name = reader.buf[reader.pos..][0..len];
                    reader.pos += len;
                },
                else => try reader.skipField(tag.wire_type),
            }
        }
        return point;
    }
};

Skipping Unknown Fields

Forward compatibility requires decoders to skip fields they don’t recognize. The wire type tells the decoder how many bytes to skip without understanding the field’s semantics:

fn skipField(reader: *ProtoReader, wire_type: WireType) !void {
    switch (wire_type) {
        .varint => { _ = try reader.readVarint(); },
        .fixed64 => { reader.pos += 8; },
        .length_delimited => {
            const len = try reader.readVarint();
            reader.pos += @intCast(len);
        },
        .fixed32 => { reader.pos += 4; },
        .start_group, .end_group => return error.DeprecatedWireType,
    }
}

Practical Applications

In a distributed vector database, for example, protobuf wire encoding is used in several critical paths:

• gRPC transport: The database’s gRPC API serializes vector search requests and results using protobuf. The custom Zig encoder avoids the overhead of protoc-generated code and enables zero-copy deserialization of vector data directly into SIMD-aligned buffers.
• On-disk storage format: Index metadata and configuration are stored as protobuf-encoded blobs in the storage engine. Protobuf’s forward compatibility means older versions of the database can read metadata written by newer versions (unknown fields are preserved on round-trip).
• Inter-node communication: Cluster nodes exchange Raft log entries, heartbeats, and replication data using protobuf over TCP. The compact encoding reduces network bandwidth, and the deterministic encoding enables content-addressable deduplication.
• Write-ahead log entries: WAL entries use a simplified protobuf encoding for point mutations, providing a compact binary representation that’s faster to write and read than JSON while remaining schema-evolvable.

Implementation Patterns

Packed Repeated Fields

For repeated numeric fields (like a vector of floats), protobuf offers “packed” encoding: instead of repeating the field tag for each element, all values are concatenated into a single length-delimited field. This saves one tag per element. For a 768-dimensional float vector, packed encoding saves 768 tags (~1.5 KB), reducing the encoded size from ~4.5 KB to ~3 KB.

Arena Allocation for Decoded Messages

Protobuf messages with repeated fields and nested messages require many small allocations during decoding. Using an arena allocator (a single large allocation that is freed all at once) avoids per-field allocation overhead and simplifies memory management. The Zig implementation decodes into arena-backed slices, and the entire decoded message is freed by resetting the arena.

Zero-Copy String and Bytes Fields

String and bytes fields in the wire format are just a length prefix followed by raw bytes. Instead of copying these bytes into a new allocation, the decoder can return a slice pointing directly into the input buffer. This is safe as long as the input buffer outlives the decoded message—a natural guarantee when the input is a memory-mapped file or a network buffer with a known lifetime.

Common Pitfalls

• Signed integer encoding: Using int32/int64 for fields that are frequently negative wastes bytes because negative two’s complement values encode as 10-byte varints. Use sint32/sint64 (zigzag encoding) for fields that may be negative, or sfixed32/sfixed64 for fields that are frequently large.
• Default value ambiguity: Protobuf 3 does not distinguish between a field set to its default value (0, "", false) and a missing field. If the difference matters, use wrapper types (google.protobuf.Int32Value) or a separate boolean “has” field.
• Field number reuse: Changing a field’s type while keeping its number silently corrupts data. If a field is removed, reserve its number with reserved 5; to prevent accidental reuse in future schema versions.
• Varint overflow: A malicious or buggy encoder could send more than 10 continuation bytes for a varint, causing the decoder to loop or overflow. Always check for overflow (shift >= 64) and limit the number of bytes consumed.
• Non-deterministic encoding: The protobuf spec does not guarantee field order, map entry order, or a unique encoding for a given message. Two semantically equal messages may produce different byte sequences. If you need deterministic encoding (for hashing or deduplication), sort fields by number and use a canonical encoding mode.