TurboQuant is an online vector quantization method for compressing high-dimensional vectors while preserving geometric operations such as reconstruction and inner-product estimation. It targets scenarios where vectors arrive continuously and cannot wait for dataset-specific training: KV cache compression, streaming embedding indexes, and low-latency vector search.
The core idea is to use a data-oblivious random rotation so coordinates behave like samples from a concentrated high-dimensional distribution, then apply scalar quantizers coordinate by coordinate. For inner products, TurboQuant adds a residual path based on Quantized Johnson-Lindenstrauss signs, because a quantizer that is good for mean-squared error can still bias dot-product estimates.
The public Zig implementation demonstrates the systems shape of the algorithm: an engine is initialized for a fixed dimension and seed, vectors are encoded into a compact binary payload, vectors can be decoded, and query vectors can be scored against compressed database vectors without fully reconstructing them first.
Many vector quantizers need a codebook trained on a representative corpus. That is effective for static indexes, but awkward for online workloads where new vectors arrive before a training pass is available. TurboQuant uses randomized preprocessing and analytic quantizers instead of learned per-dataset codebooks, so the encoder can compress each vector independently.
Random rotation spreads vector energy across coordinates. After rotation, coordinate values in high dimension become easier to model with a shared scalar quantizer. This avoids treating raw coordinate axes as meaningful, which is often a poor assumption for embedding vectors and transformer activations.
A reconstruction-oriented quantizer can minimize squared error while still producing biased dot products. TurboQuant addresses this by adding a one-bit Quantized Johnson-Lindenstrauss residual component. The residual signs help correct inner-product estimates without requiring a full high-precision decode of the stored vector.
ANN systems care about ranking, not only decompression quality. TurboQuant treats MSE and dot product distortion as related but distinct objectives, then adds residual information for the scoring path.
A production implementation usually exposes an engine with fixed dimensionality, deterministic random state, and reusable buffers. The repository API follows that shape: initialize once, encode many vectors, and score query vectors directly against compressed payloads.
const turboquant = @import("turboquant");
var engine = try turboquant.Engine.init(allocator, .{
.dim = 1024,
.seed = 12345,
});
defer engine.deinit(allocator);
const compressed = try engine.encode(allocator, vector);
defer allocator.free(compressed);
const approx = try engine.decode(allocator, compressed);
defer allocator.free(approx);
const score = engine.dot(query_vector, compressed);The implementation stores metadata beside the compressed vector so decode and dot-product paths know which components are present. The documented format uses a small header with version, dimension, component sizes, radius, and quantizer scale, followed by polar-encoded and QJL payload sections.
Header:
version: u8
dim: u32
polar_bytes: u32
qjl_bytes: u32
max_r: f32
gamma: f32
Payload:
polar encoded component
QJL residual componentANN indexes can store compressed database vectors and estimate query similarities before fetching or reconstructing full-precision vectors. This reduces memory bandwidth and cache pressure, especially when candidate sets are large and ranking uses many dot products.
Transformer key and value caches grow linearly with sequence length and layer count. A low-bit online quantizer can compress newly produced cache vectors as decoding proceeds, without retraining the model or building a corpus-specific codebook. The paper reports strong long-context behavior at low bit widths for this use case.
Systems that ingest embeddings continuously need compression that can be applied as vectors arrive. Data-oblivious online quantization fits ingestion pipelines where vectors must become searchable immediately and later rebuilds are expensive.
The encoder and scorer must agree on dimension, seed, quantizer parameters, and payload version. Store those values in index metadata and in every shard manifest. If a seed changes, dot-product estimates become meaningless even if byte lengths still look valid.
type QuantizedVector struct {
Version uint8
Dimension uint32
Seed uint32
PolarBytes []byte
QJLBytes []byte
MaxRadius float32
Gamma float32
}Use compressed scoring to rank a broad candidate set, then optionally re-score the final window with full-precision vectors if they are available. This pattern preserves most of the bandwidth savings while bounding quality risk for top-k results.
The dot path is the hot path. Store compressed payloads in aligned batches, avoid per-vector heap allocation during search, and specialize kernels for the target CPU when possible. The Zig implementation’s use of portable vector operations is a practical example of keeping the public API stable while allowing hardware-specific acceleration underneath.
k, candidate-pool size, and embedding distribution rather than relying only on average MSE.