Database benchmarking is the systematic measurement of a database system’s performance under controlled conditions. Unlike ad-hoc “how fast is this query?” testing, rigorous benchmarking produces reproducible, comparable results that inform engineering decisions: which index configuration to deploy, whether a query engine optimization actually helps under production-like load, and how the system performs relative to competing solutions. For vector databases, benchmarking is especially critical because the fundamental tradeoff — search quality (recall) versus search speed (latency) — requires precise measurement to navigate.
A well-structured benchmarking framework built on VectorDBBench patterns measures recall, nDCG (normalized discounted cumulative gain), MRR (mean reciprocal rank), throughput (queries per second), and latency distributions (p50, p95, p99) across varying dataset sizes, query patterns, and search-effort parameters. Results are compared against competing vector databases — Qdrant, Milvus, Chroma — to quantify the system’s positioning on the accuracy-speed Pareto frontier.
This topic covers the principles of sound database benchmarking, the specific metrics used for vector search evaluation, framework design for reproducible benchmarks, statistical analysis of results, and the search-effort tuning methodology that allows users to trade precision for speed based on their application requirements.
Recall measures the fraction of true nearest neighbors that the search algorithm actually returns. If the ground truth for a query is 10 specific vectors and the algorithm returns 8 of them, recall is 0.80. For approximate nearest neighbor (ANN) search, recall is the primary quality metric because ANN algorithms trade exactness for speed. A recall of 0.99 means the algorithm misses 1 in 100 true neighbors — acceptable for recommendation systems but potentially unacceptable for medical or legal retrieval. Benchmarking must measure recall at every operating point to map the accuracy-speed tradeoff curve.
Normalized Discounted Cumulative Gain measures not just whether the correct results are returned but whether they’re returned in the right order. Results at the top of the list are weighted more heavily than those lower down, reflecting real-world usage where users primarily examine the first few results. nDCG ranges from 0 to 1, where 1 means perfect ranking. For vector search, nDCG captures whether the closest vectors are ranked highest — a subtler signal than binary recall, especially important for search and recommendation applications where result ordering directly affects user experience.
Average latency is misleading for database benchmarks because it hides tail behavior. A system with 2ms average latency might have p99 at 200ms — meaning 1 in 100 queries takes 100x longer than typical. Percentile measurements (p50 = median, p95 = 95th percentile, p99 = 99th percentile) characterize the full latency distribution. For production systems, p99 latency determines SLA compliance and user experience during load spikes. Benchmarks should report at minimum p50, p95, and p99, measured under sustained load rather than single-query conditions.
Most ANN algorithms expose a tunable “search effort” parameter that controls how thoroughly the algorithm explores the index during query time. In HNSW this is ef_search; in IVF indices it’s nprobe. Higher values increase recall but also increase latency. Benchmarking must sweep this parameter across a range of values to produce recall-vs-latency curves rather than single-point measurements. The optimal setting depends on the application: a chatbot may tolerate 0.95 recall for 1ms latency, while a legal discovery system needs 0.999 recall regardless of cost.
A benchmark result is only useful if someone else can reproduce it. This requires controlling: the dataset (exact vectors and queries), the hardware (CPU model, RAM, disk type, core count), the software version (exact commit hash), the system state (cold start vs. warm cache), and the load pattern (concurrent clients, request rate). Without these controls, benchmark numbers are anecdotes, not evidence. Containerized benchmark runners with pinned dependencies and hardware-specific profiles ensure that results from different dates or machines are comparable.
A benchmarking run proceeds through four phases: dataset preparation, system configuration, workload execution, and result analysis. Each phase must be deterministic and documented.
// benchmark_config.go — configuration for a benchmark run
type BenchmarkConfig struct {
// Dataset configuration
Dataset string `json:"dataset"` // e.g., "sift-1m", "gist-960"
Dimensions int `json:"dimensions"` // vector dimensionality
NumVectors int `json:"num_vectors"` // dataset size
NumQueries int `json:"num_queries"` // query set size
K int `json:"k"` // top-K results to retrieve
GroundTruthURL string `json:"ground_truth_url"`// pre-computed exact results
// System under test
Systems []SystemConfig `json:"systems"`
// Search effort sweep — each value produces one data point
// on the recall-vs-latency curve.
SearchEfforts []int `json:"search_efforts"` // e.g., [10, 20, 50, 100, 200, 500]
// Load configuration
Concurrency int `json:"concurrency"` // parallel query clients
WarmupQueries int `json:"warmup_queries"` // queries before measurement
Duration time.Duration `json:"duration"` // measurement window
}
type SystemConfig struct {
Name string `json:"name"` // e.g., "exampledb", "qdrant"
Version string `json:"version"` // exact version or commit hash
Params map[string]any `json:"params"` // index build parameters
}// metrics.go — computing recall against ground truth
package benchmark
// ComputeRecall calculates recall@K: the fraction of true top-K
// neighbors that appear in the predicted results.
func ComputeRecall(predicted, groundTruth []uint64, k int) float64 {
if k > len(groundTruth) {
k = len(groundTruth)
}
// Build a set of the true top-K neighbors.
truthSet := make(map[uint64]struct{}, k)
for _, id := range groundTruth[:k] {
truthSet[id] = struct{}{}
}
// Count how many predicted results are in the truth set.
hits := 0
for _, id := range predicted {
if _, ok := truthSet[id]; ok {
hits++
}
}
return float64(hits) / float64(k)
}
// ComputeNDCG calculates normalized discounted cumulative gain,
// measuring ranking quality where higher-ranked correct results
// score more than lower-ranked ones.
func ComputeNDCG(predicted, groundTruth []uint64, k int) float64 {
relevance := make(map[uint64]float64, len(groundTruth))
for i, id := range groundTruth {
// Relevance decreases with ground-truth rank.
relevance[id] = float64(len(groundTruth) - i)
}
// DCG: sum of relevance / log2(rank + 1)
dcg := 0.0
for i, id := range predicted[:min(k, len(predicted))] {
if rel, ok := relevance[id]; ok {
dcg += rel / math.Log2(float64(i+2)) // i+2 because rank is 1-indexed
}
}
// Ideal DCG: ground truth in perfect order.
idcg := 0.0
for i := 0; i < min(k, len(groundTruth)); i++ {
rel := float64(len(groundTruth) - i)
idcg += rel / math.Log2(float64(i+2))
}
if idcg == 0 {
return 0
}
return dcg / idcg
}// latency.go — precise latency tracking using HDR histograms
package benchmark
import (
"time"
"github.com/HdrHistogram/hdrhistogram-go"
)
type LatencyTracker struct {
hist *hdrhistogram.Histogram
}
func NewLatencyTracker() *LatencyTracker {
// Track latencies from 1 microsecond to 10 seconds
// with 3 significant digits of precision.
return &LatencyTracker{
hist: hdrhistogram.New(1, 10_000_000, 3),
}
}
func (lt *LatencyTracker) Record(d time.Duration) {
lt.hist.RecordValue(d.Microseconds())
}
type LatencyReport struct {
P50 time.Duration `json:"p50"`
P95 time.Duration `json:"p95"`
P99 time.Duration `json:"p99"`
P999 time.Duration `json:"p999"`
Mean time.Duration `json:"mean"`
Max time.Duration `json:"max"`
}
func (lt *LatencyTracker) Report() LatencyReport {
return LatencyReport{
P50: time.Duration(lt.hist.ValueAtQuantile(50)) * time.Microsecond,
P95: time.Duration(lt.hist.ValueAtQuantile(95)) * time.Microsecond,
P99: time.Duration(lt.hist.ValueAtQuantile(99)) * time.Microsecond,
P999: time.Duration(lt.hist.ValueAtQuantile(99.9)) * time.Microsecond,
Mean: time.Duration(lt.hist.Mean()) * time.Microsecond,
Max: time.Duration(lt.hist.Max()) * time.Microsecond,
}
}// sweep.go — sweeping search effort to map the recall-latency curve
func RunSearchEffortSweep(
client DBClient,
queries [][]float32,
groundTruth [][]uint64,
efforts []int,
k int,
) []SweepResult {
results := make([]SweepResult, 0, len(efforts))
for _, effort := range efforts {
client.SetSearchEffort(effort)
tracker := NewLatencyTracker()
totalRecall := 0.0
totalNDCG := 0.0
for i, query := range queries {
start := time.Now()
predicted, err := client.Search(query, k)
elapsed := time.Since(start)
if err != nil {
log.Printf("query %d failed: %v", i, err)
continue
}
tracker.Record(elapsed)
totalRecall += ComputeRecall(predicted, groundTruth[i], k)
totalNDCG += ComputeNDCG(predicted, groundTruth[i], k)
}
n := float64(len(queries))
results = append(results, SweepResult{
SearchEffort: effort,
AvgRecall: totalRecall / n,
AvgNDCG: totalNDCG / n,
Latency: tracker.Report(),
QPS: n / tracker.Report().Mean.Seconds() / n,
})
}
return results
}A typical benchmarking project runs comparative benchmarks across standard ANN benchmark datasets (SIFT-1M, GIST-960, GloVe-200) to position a vector database candidate against competing systems like Qdrant, Milvus, and Chroma. Each system is configured with equivalent index parameters (HNSW with matching M and ef_construction values), loaded with identical datasets, and queried with the same query vectors. The output is a set of recall-vs-latency Pareto curves that show, for any target recall level, which system delivers the lowest latency — and by how much.
These benchmarks run nightly in CI on dedicated bare-metal machines (to avoid cloud VM noisy-neighbor effects) and produce a dashboard of historical performance trends. When a code change regresses p99 latency by more than 10% or recall by more than 0.1%, the CI pipeline flags it as a performance regression and blocks the merge. This automated performance gating catches optimization regressions that unit tests cannot detect — a query planner change might be functionally correct but 3x slower on high-dimensional datasets.
The search-effort tuning results are also published as user-facing documentation: a table showing recommended ef_search values for different recall targets (0.95, 0.99, 0.999) on standard dataset profiles. This gives operators concrete guidance rather than requiring them to run their own benchmarks to find the right configuration for their latency and accuracy requirements.
Database systems cache data structures, JIT-compile query paths, and warm OS page caches during initial queries. Benchmarking without a warm-up phase measures cold-start behavior rather than steady-state performance. Always run a configurable number of warm-up queries (typically 10-20% of the measurement queries) before starting latency measurement. Discard warm-up results entirely — don’t average them with steady-state numbers.
Recall measurement requires knowing the exact top-K neighbors for each query. For small datasets, compute ground truth with brute-force exact search. For large datasets where brute-force is prohibitive, use a trusted implementation (FAISS with IndexFlat) and verify the ground truth against a second independent implementation. Store ground truth alongside the dataset so it doesn’t need recomputation for every benchmark run.
Run each benchmark configuration multiple times (at least 3 runs) and report the median with confidence intervals. A single run can be skewed by OS scheduling, thermal throttling, or background processes. If two systems’ confidence intervals overlap, the performance difference is not statistically significant — don’t claim one is faster based on noise. For competitive benchmarks, use paired comparisons on the same hardware to control for machine-specific variance.
If the same query is repeated, database caches (OS page cache, internal result caches, CPU cache) make subsequent executions artificially fast. Real workloads rarely repeat the exact same query. Use a query set large enough that the working set exceeds cache capacity, or explicitly flush caches between iterations. Better yet, use query logs from production to benchmark with realistic access patterns.
Vector search benchmarks often focus exclusively on query latency and ignore indexing throughput. But ingestion speed matters: if a system takes 4 hours to index 10M vectors while a competitor takes 20 minutes, the slower system blocks development iteration and delays production data refreshes. Always benchmark both ingestion throughput (vectors/second) and query performance, and report them together.
Benchmarking system A on an m5.2xlarge and system B on a c6g.4xlarge produces meaningless comparisons. Even with the same instance type, cloud VMs have variable performance due to noisy neighbors and CPU throttling. For competitive benchmarks, run all systems on the same physical machine in sequence (not simultaneously). For CI regression testing, use dedicated bare-metal or reserved instances with consistent performance characteristics.
A benchmark that sends one query at a time measures single-thread latency, not system throughput. Real workloads are concurrent. Benchmark with realistic concurrency levels (8, 16, 32, 64 parallel clients) and measure both throughput (QPS) and per-query latency under load. Many systems have dramatically different performance characteristics under concurrency — a system with excellent single-query latency may collapse under 32 concurrent queries due to lock contention.