Snappy and LZ4 occupy the fast tier of general-purpose lossless compression. They intentionally sacrifice some compression ratio so storage engines, message brokers, and RPC systems can compress hot data without turning CPU into the bottleneck. DDIA introduces compression in the context of storage files, network traffic, and column-oriented layouts; the practical systems question is not simply “should this data be compressed?” but “which codec fits this latency budget?”
Google’s Snappy README states the design goal plainly: it does not target maximum density or format compatibility; it targets very high speed and reasonable compression. Its documented single-core benchmark on a 64-bit Core i7 reports about 250 MB/s or more for compression and about 500 MB/s or more for decompression on the slowest benchmark inputs, while producing outputs roughly 20% to 100% larger than zlib’s fastest mode. That is exactly the trade many databases accept for write path blocks, replication frames, and cache-resident files.
LZ4 pushes the same family of trade-offs even harder. The LZ4 project describes compression speed above 500 MB/s per core and decompression in multiple GB/s per core, often reaching memory bandwidth limits. It also exposes an acceleration factor for lower ratio and higher speed, plus LZ4_HC for higher ratio at the same decompression speed. In a storage engine, this gives operators a simple mental model: Snappy and regular LZ4 are hot-path codecs; LZ4_HC and Zstd are colder-path codecs.
The important architectural point is that fast compression is a latency primitive. A RocksDB block compressed with Snappy or LZ4 must be cheap to encode during flush or compaction, cheap to decode during reads, and stable across rolling upgrades. The codec must be predictable enough that p99 read latency does not swing wildly when cache misses require decompression.
Both Snappy and LZ4 are dictionary-free, byte-oriented compressors in the LZ77 family. They scan the input, copy literal bytes when no useful match is found, and emit back references when a later byte sequence repeats an earlier byte sequence. The compressed stream is therefore a sequence of literal runs and copy instructions. Decompression is fast because it does not search for matches; it only copies literals and resolves offsets against already decoded bytes.
Fast codecs are usually deployed at block granularity: a 4 KB, 16 KB, 32 KB, or 64 KB database block is compressed independently and tagged with its codec. Independent blocks preserve random access. A point lookup reads the index, fetches one compressed block, decompresses it, and searches inside the uncompressed block. This is weaker than whole-file compression for ratio, but it is the right trade-off for storage engines.
Compression requires match finding, so it is slower and more branchy. Decompression only interprets the encoded stream, so it is usually faster and more memory-bandwidth bound. LZ4’s regular, accelerated, and high-compression variants all keep the same decompression speed profile, which is why a system can spend extra CPU compressing cold data without penalizing readers later.
A hot-path codec should be chosen by bytes saved per microsecond of CPU, not by compression ratio alone. If compressing a block saves 20 microseconds of disk or network time but costs 80 microseconds of CPU on a saturated core, the codec has moved the bottleneck instead of removing it. Snappy and LZ4 are useful because they often save enough bytes while keeping CPU cost inside request latency budgets.
A simplified fast LZ encoder keeps a hash table from short byte sequences to their most recent positions. At each input offset, it hashes the next four or eight bytes, checks the previous position for a match, and either emits a literal byte or a copy token. Real Snappy and LZ4 formats differ in token layout, limits, checks, and acceleration rules, but the control flow below captures the core mechanism.
type Token struct {
Kind string // "literal" or "copy"
Bytes []byte
Offset int
Length int
}
func encodeFastLZ(src []byte) []Token {
table := map[uint32]int{}
out := []Token{}
literals := []byte{}
i := 0
for i < len(src) {
if i+4 > len(src) {
literals = append(literals, src[i])
i++
continue
}
h := hash4(src[i : i+4])
prev, ok := table[h]
table[h] = i
if ok && i-prev < 1<<16 && equal4(src[prev:prev+4], src[i:i+4]) {
if len(literals) > 0 {
out = append(out, Token{Kind: "literal", Bytes: literals})
literals = nil
}
n := extendMatch(src, prev, i)
out = append(out, Token{Kind: "copy", Offset: i - prev, Length: n})
i += n
continue
}
literals = append(literals, src[i])
i++
}
if len(literals) > 0 {
out = append(out, Token{Kind: "literal", Bytes: literals})
}
return out
}The decoder is intentionally simpler. Literal tokens append bytes directly. Copy tokens read bytes from the already decoded output at a backward offset. Overlapping copies are legal in LZ-family codecs, so the implementation must copy byte by byte or use a carefully written vectorized loop that preserves overlap semantics.
func decodeFastLZ(tokens []Token) []byte {
out := []byte{}
for _, t := range tokens {
switch t.Kind {
case "literal":
out = append(out, t.Bytes...)
case "copy":
start := len(out) - t.Offset
for j := 0; j < t.Length; j++ {
out = append(out, out[start+j])
}
}
}
return out
}The codec stream is usually wrapped in a storage-engine frame. The frame records the uncompressed length, compressed length, codec id, checksum, and optional dictionary or version metadata. This frame is not the same thing as the codec’s public file format; it is a local container that lets the engine skip decompression for uncompressed blocks, validate corruption before handing bytes to the codec, and choose a decoder at runtime.
RocksDB defaults to Snappy for compatibility but explicitly notes that LZ4 is almost always better, leaving Snappy as the default to avoid unexpected compatibility breaks for existing users. The same page recommends using lightweight compression such as LZ4 or Snappy for ordinary levels, then heavier Zstd or zlib for the bottommost level where most data lives and write amplification has already cooled down. That two-tier pattern maps directly to LSM storage economics.
Kafka, Cassandra, ScyllaDB, and similar systems often compress batches rather than individual records. A batch gives the codec enough repeated structure to find matches while keeping decompression scoped to the data a consumer or replica already needs. Fast decompression matters because consumers may fan out across many partitions, and a high-ratio codec can become a per-core throttle even when network bytes drop.
Columnar formats frequently layer encodings: dictionary, run-length, delta, bit-packing, then a block codec. Snappy and LZ4 are good final-stage codecs when the first-stage encodings already removed much of the redundancy and the remaining goal is to reduce I/O without slowing scans. For highly repetitive cold columns, Zstd may win; for hot scans, LZ4 often wins because decompression is close to memory copy speed.
Do not store a block compressed if the result is barely smaller. Many engines use a threshold: if compressed bytes are not at least, say, 12.5% smaller than raw bytes, store the block uncompressed and mark the frame accordingly. This avoids wasting CPU and sometimes avoids read amplification on incompressible payloads such as JPEGs, encrypted blobs, or already-compressed model artifacts.
type Codec interface {
ID() byte
Compress(dst, src []byte) ([]byte, error)
Decompress(dst, src []byte, rawLen int) ([]byte, error)
}
type BlockFrame struct {
CodecID byte
RawLength uint32
StoredLength uint32
Checksum uint32
Payload []byte
}
func encodeBlock(raw []byte, codec Codec, minSavingsPct int) (BlockFrame, error) {
compressed, err := codec.Compress(nil, raw)
if err != nil {
return BlockFrame{}, err
}
required := len(raw) * (100 - minSavingsPct) / 100
if len(compressed) >= required {
return BlockFrame{
CodecID: 0, // raw
RawLength: uint32(len(raw)),
StoredLength: uint32(len(raw)),
Checksum: crc32(raw),
Payload: append([]byte(nil), raw...),
}, nil
}
return BlockFrame{
CodecID: codec.ID(),
RawLength: uint32(len(raw)),
StoredLength: uint32(len(compressed)),
Checksum: crc32(raw),
Payload: compressed,
}, nil
}Network protocols should negotiate codecs explicitly. A sender must not assume every peer supports LZ4 just because the current cluster does. Store a codec capability bitmap in handshake metadata and make the frame self-describing. For storage files, persist a stable codec id and reject unsupported ids with a clear error rather than attempting best-effort decoding.
Even fast codecs allocate scratch buffers and decoder state. A production engine usually pools encoder and decoder contexts per codec, per compression level, and sometimes per block size. Pooling is most important when many goroutines are compressing small blocks, because allocation and garbage collection can dominate the codec’s actual CPU cost.