Compression strategy is a systems design problem, not a library checkbox. Snappy, LZ4, Zstd, Brotli, and gzip all reduce bytes, but they optimize different parts of the design space: latency, ratio, decompression speed, dictionary support, ecosystem compatibility, streaming APIs, and operational predictability. A storage engineer choosing one codec for SSTables, one for network frames, and one for archives is really allocating CPU, memory bandwidth, disk, and compatibility risk.
Zstd was created to widen the historical speed-ratio frontier. Gregory Szorc’s analysis highlights the practical reasons this matters: benchmark results vary by corpus, but Zstd gives engineers a broad set of levels, a strong C API, and dictionary compression that can materially improve small-record workloads. In one Firefox omni.ja experiment, a trained 128 KiB dictionary reduced discrete-file compressed size by more than 1.1 MB compared with non-dictionary Zstd, and dictionary decompression over Mercurial changeset data improved from roughly 240 MB/s to roughly 530 MB/s.
The other codecs still matter. Snappy and LZ4 are often better for hot-path database blocks because they keep latency and CPU variance low. gzip remains the compatibility baseline for HTTP, archives, and legacy tooling. Brotli targets high-density web delivery and static assets, with an RFC-defined format built from LZ77, Huffman coding, and second-order context modeling. There is no universal winner; the right answer depends on where the bytes sit in the system and who pays the CPU bill.
Snappy and LZ4 are latency-tier codecs. They are appropriate when compression and decompression happen on the request path or replication path. Zstd spans both tiers: low levels behave like fast storage codecs, while high levels compete with archive codecs. gzip and Brotli are ratio-tier or compatibility-tier choices: gzip because every platform understands it, Brotli because web delivery and static bundles often value transfer size more than compression CPU.
Many workloads write once and read many times. A backup job or bottommost LSM compaction can afford slower compression if every future read becomes smaller and decompression remains fast. A pub/sub producer, by contrast, pays compression cost for every batch under live traffic. A codec decision should therefore weight compression and decompression separately instead of treating “speed” as one scalar.
Small records have little local history, so ordinary stream compression often underperforms. Dictionary compression seeds the compressor and decompressor with representative patterns. Zstd, Brotli, and zlib support dictionary-style use cases, while Snappy and ordinary LZ4 are usually deployed without trained dictionaries. Dictionaries improve ratio only when training samples match production data and when producers and consumers can reliably distribute the same dictionary version.
If the bottleneck is CPU, choose a faster codec or store hot data uncompressed. If the bottleneck is disk or network, choose a denser codec. If the bottleneck is compatibility, choose gzip or a format the ecosystem already supports. If the bottleneck is random access, use independent compressed blocks rather than a monolithic stream.
A defensible codec choice starts with a workload model. Estimate how many bytes are compressed, how many are decompressed, how expensive storage or transfer is, and how much CPU time each codec consumes. The score does not need to be perfect; it needs to prevent ratio-only decisions that harm latency-sensitive paths.
type CodecBench struct {
Name string
Ratio float64 // uncompressed bytes / compressed bytes
CompressMBPerSec float64
DecompressMBPerSec float64
}
type Workload struct {
InputMiB float64
CompressesPerDay float64
DecompressesPerDay float64
StorageCostPerMiBDay float64
NetworkCostPerMiB float64
CpuCostPerSecond float64
}
func estimateDailyCost(c CodecBench, w Workload) float64 {
compressedMiB := w.InputMiB / c.Ratio
savedMiB := w.InputMiB - compressedMiB
compressCPU := (w.InputMiB / c.CompressMBPerSec) * w.CompressesPerDay
decompressCPU := (w.InputMiB / c.DecompressMBPerSec) * w.DecompressesPerDay
storage := compressedMiB * w.StorageCostPerMiBDay
network := compressedMiB * w.NetworkCostPerMiB
cpu := (compressCPU + decompressCPU) * w.CpuCostPerSecond
// savedMiB is useful for reporting even if the objective is total cost.
_ = savedMiB
return storage + network + cpu
}For foreground database reads and writes, start with LZ4 or Snappy, then test low-level Zstd if ratio pressure is real. For cold LSM levels, backups, artifacts, and long-lived archives, start with Zstd and evaluate higher levels if compression runs offline. For HTTP text assets, compare Brotli and gzip because browser support and CDN behavior dominate. For legacy file exchange, gzip may be the only acceptable answer despite weaker density.
Block compression is the default for indexed storage and random access. Stream compression is the default for logs, archives, HTTP responses, and bulk transfer. A stream codec can produce excellent ratio because it carries history across a long input, but random reads may require decompressing from the beginning or from a seekable checkpoint. A block codec gives up history to keep seeks cheap.
An LSM tree can use multiple codecs at once. Memtable flushes and upper levels favor LZ4 or Snappy because they are rewritten frequently and read by foreground traffic. Bottommost levels favor Zstd because they contain most of the data and change less often. This is the RocksDB pattern: lightweight compression for normal levels, heavier bottommost compression when CPU is available.
Web servers commonly serve gzip for broad compatibility and Brotli for clients that support it. Brotli’s higher levels are expensive enough that they are best for precompressed static assets, not dynamic per-request payloads. Zstd-over-HTTP is increasingly interesting, but deployment depends on client and proxy support; codec selection must include the whole delivery path, not only the origin server.
Telemetry events, WAL records, and API payloads are often small and structurally similar. Zstd dictionaries can be a major win here, but only if dictionary lifecycle is engineered: training, versioning, rollout, fallback, monitoring, and retirement. Without that operational layer, dictionary compression can turn a data-saving feature into an incident source.
Represent compression choices as policy, not scattered constants. Each storage tier or transport should declare its latency goal, default codec, accepted fallbacks, minimum savings threshold, and maximum decompressed size. This keeps upgrades reviewable and prevents one team’s archive setting from leaking into another team’s hot path.
type CompressionPolicy struct {
Tier string
DefaultCodec string
FallbackCodec string
MinSavingsPct int
MaxDecodedBytes int
AllowsDictionary bool
}
var Policies = map[string]CompressionPolicy{
"wal": {
Tier: "foreground-log", DefaultCodec: "lz4", FallbackCodec: "raw",
MinSavingsPct: 10, MaxDecodedBytes: 1 << 20, AllowsDictionary: false,
},
"sstable-upper": {
Tier: "hot-storage", DefaultCodec: "lz4", FallbackCodec: "snappy",
MinSavingsPct: 12, MaxDecodedBytes: 64 << 10, AllowsDictionary: false,
},
"sstable-bottom": {
Tier: "cold-storage", DefaultCodec: "zstd", FallbackCodec: "lz4",
MinSavingsPct: 5, MaxDecodedBytes: 256 << 10, AllowsDictionary: true,
},
"static-web": {
Tier: "edge-delivery", DefaultCodec: "brotli", FallbackCodec: "gzip",
MinSavingsPct: 1, MaxDecodedBytes: 16 << 20, AllowsDictionary: false,
},
}Dictionary-compressed frames need a dictionary id, a codec id, and a safe fallback behavior when the reader lacks the dictionary. A rolling deployment should publish the dictionary to all readers before writers start emitting frames that require it. Metrics should track dictionary hit rate, fallback rate, compression ratio, and decode failures by version.
Benchmark with the same block sizes, data distributions, CPU limits, and concurrency that production uses. Include incompressible samples and adversarial small samples. Report compression speed, decompression speed, ratio, p50 and p99 latency, allocation rate, and total end-to-end time with I/O or network included. The Squash benchmark is useful because it exposes raw data and separates ratio, compression speed, decompression speed, and transfer-plus-processing views.
omni.ja case where a 128 KiB trained dictionary made discrete-file Zstd output more than 1.1 MB smaller than non-dictionary Zstd, including the dictionary cost.