Vector retrieval systems often need to solve maximum inner-product search (MIPS): given a query embedding, find the dataset embeddings with the highest dot product. That is the core retrieval primitive behind semantic search, candidate generation in recommenders, and two-tower retrieval models. The difficulty is not defining the metric, but making it fast enough at million- to billion-vector scale without scanning everything.
ScaNN, released by Google Research, is a practical answer to that problem. It combines three ideas into one pipeline: partitioning to prune most of the dataset, asymmetric hashing to score candidates cheaply, and rescoring to recover accuracy on the final shortlist. Its signature algorithmic contribution is the use of anisotropic vector quantization (AVQ), a quantization objective optimized for preserving top inner-product rankings rather than minimizing reconstruction error.
That distinction matters. Standard vector quantization is usually trained to make the compressed vector close to the original vector in Euclidean distance. ScaNN’s AVQ instead treats error in the direction parallel to the original vector as more damaging than orthogonal error, because ranking mistakes in MIPS come primarily from distorting the largest inner products. The result, according to Google’s paper and benchmark report, is a system that can deliver roughly 2x higher queries-per-second at a given recall than competing libraries on the referenced ANN benchmark configuration.
This topic covers ScaNN as an ANN design pattern, not just as a library install. The real lesson is how to align compression loss, pruning strategy, and re-ranking work with the retrieval objective you actually care about.
In MIPS, the score for candidate vector x under query q is <q, x>. This is common when embeddings are used as learned retrieval keys, especially in two-tower architectures where queries and documents are embedded independently. MIPS differs from plain Euclidean nearest-neighbor search because preserving the ordering of top dot products matters more than reproducing exact vector coordinates.
AVQ changes the quantization objective. Instead of minimizing average reconstruction distortion, it penalizes residual error that is parallel to the datapoint more heavily than error that is orthogonal to it. The Google Research blog illustrates why: a quantized point that is farther away in Euclidean terms can still preserve the correct ranking if its error stays mostly orthogonal to the vector direction relevant for inner-product search.
ScaNN’s system design is explicitly staged. Optional tree partitioning narrows the search to a subset of leaves. Scoring uses brute force or asymmetric hashing over the surviving candidates. Optional rescoring recomputes more accurate distances for the best k’ candidates before returning the final top-k. This structure is what makes the library tunable across dataset size, recall target, and latency budget.
The central ScaNN idea is that ANN compression should preserve the ordering of the best neighbors, not the average geometric fidelity of all vectors. AVQ is valuable because it spends approximation error where ranking is least sensitive and protects the high-score region where mistakes are costly.
The ScaNN documentation describes the runtime as three phases: partitioning, scoring, and rescoring. For large datasets, the index is trained with a tree of leaves; at query time the search visits only the top leaves, scores all points inside those leaves approximately, and then reranks a smaller candidate set more accurately. This keeps full-precision work off the hot path.
import numpy as np
import scann
normalized_dataset = dataset / np.linalg.norm(dataset, axis=1)[:, np.newaxis]
searcher = scann.scann_ops_pybind.builder(
normalized_dataset, 10, "dot_product"
).tree(
num_leaves=2000,
num_leaves_to_search=100,
training_sample_size=250000
).score_ah(
2, anisotropic_quantization_threshold=0.2
).reorder(
100
).build()This example from ScaNN’s notebook shows the system-level composition directly. The tree stage prunes the corpus, score_ah enables asymmetric hashing with anisotropic quantization, and reorder(100) rescans the best 100 candidates before returning the final 10 neighbors.
In the AVQ paper, Google formalizes a family of anisotropic loss functions that weight the parallel component of quantization residual more strongly than the orthogonal component. Intuitively, if a residual shifts a vector mostly along its own direction, it perturbs the inner product with aligned queries more severely. A mostly orthogonal residual may increase reconstruction error while leaving the top-k ranking comparatively intact.
That is why ScaNN can prefer a centroid that is not the nearest in Euclidean distance if that choice better preserves the query-time ordering induced by dot products. The search system is biased toward better top-neighbor estimation, not prettier compression statistics.
Text retrieval systems built on embedding models routinely index hundreds of thousands to millions of passages. ScaNN’s documented guidance is to use brute force for very small datasets, asymmetric hashing for mid-sized datasets, and tree partitioning plus asymmetric hashing plus rescoring once the corpus exceeds roughly 100k points. That makes it a good fit for large semantic-search backends where recall targets are strict but exhaustive scan is too slow.
The Google Research post explicitly motivates ScaNN with two-tower retrieval models. In that setting, the online tower embeds the request, the offline tower embeds the inventory, and MIPS is the bottleneck. A partitioned approximate index can cut candidate-generation latency enough to make retrieval-based recommenders practical at serving time.
The ScaNN algorithms guide gives concrete rules of thumb that are more useful than generic ANN folklore: brute force below roughly 20k points, AH + rescore below roughly 100k points, and tree + AH + rescore beyond that. It also recommends setting dimensions_per_block to 2 when using AH and choosing num_leaves on the order of sqrt(n).
# Typical tuning directions from the ScaNN docs:
# - num_leaves: roughly sqrt(n)
# - num_leaves_to_search: trade recall for latency
# - dimensions_per_block: 2 for AH
# - reordering_num_neighbors: larger than final k
searcher = scann.scann_ops_pybind.builder(data, k, "dot_product").tree(
num_leaves=int(np.sqrt(len(data))),
num_leaves_to_search=leaf_budget
).score_ah(
2, anisotropic_quantization_threshold=0.2
).reorder(
reordering_num_neighbors
).build()ScaNN works best when approximate work is cheap and plentiful while exact work is narrow and selective. That means using partitioning to shrink the candidate pool, compressed scoring to cheaply rank the remainder, and rescoring only for a shortlist whose size you can afford to touch with more accurate arithmetic. The architectural pattern generalizes beyond ScaNN: prune aggressively, approximate broadly, verify narrowly.
num_leaves_to_search / num_leaves controls the prune ratio. Searching too few leaves hurts recall; searching too many collapses back toward a scan.reordering_num_neighbors usually improves accuracy, but it directly increases exact-distance work on the tail of each query.num_leaves, num_leaves_to_search, and reorder width all depend on corpus size and recall requirements. There is no single optimal profile.