Search aggregations compute analytics over the documents that match a query, transforming a search engine from a retrieval tool into an analytics platform. Instead of returning individual documents, aggregations answer questions like “what is the average price of matching products?”, “how are results distributed across date ranges?”, or “where are matching locations clustered on a map?”
Aggregations fall into three broad families: metric aggregations that compute statistics (sum, average, min, max, percentiles), bucket aggregations that partition documents into groups (term frequency, date histograms, numeric ranges, geohash grids), and pipeline aggregations that chain the output of one aggregation as input to another. Most search engines support nesting these arbitrarily—a date histogram bucketed by month, with a terms sub-aggregation by category, with an average price metric inside each category bucket.
In a typical search engine implementation, aggregations are built as a collector layer that runs alongside query evaluation. As the scorer iterates matching documents, each aggregation collector receives the document ID and extracts the values it needs from column-oriented stored fields. This architecture avoids a second pass over results and keeps aggregation latency proportional to the number of matching documents rather than the total index size.
Compute a single numeric value over matching documents: sum, average, min, max, count, cardinality (approximate via HyperLogLog), and percentiles (via t-digest or HDR histogram). These are leaf aggregations—they produce a number, not sub-buckets.
Partition matching documents into groups (buckets) based on field values. Each bucket acts as a sub-query scope. Terms aggregation groups by exact field value; histogram and date_histogram group by numeric or time intervals; range aggregation groups by explicit ranges; geohash_grid groups by geographic cell.
Aggregations are implemented as collectors that receive document-hit callbacks during query evaluation. Each collector maintains its own state (running sums, bucket maps, sketches). This single-pass design means aggregations add marginal cost on top of the query itself rather than requiring a separate scan.
Aggregations need fast access to field values for every matching document. Column stores (doc values) store each field as a contiguous array indexed by document ID, enabling sequential reads with high cache locality—critical when aggregating over millions of documents.
Statistical aggregations compute descriptive statistics over numeric fields. The simplest are sum, min, max, and average, which require O(1) state per aggregation. More advanced statistics include variance, standard deviation (computed via Welford’s online algorithm to avoid numerical instability), and percentiles. Percentile estimation uses streaming algorithms because exact computation requires sorting all values.
Histogram aggregations create fixed-width buckets over a numeric or date field. A numeric histogram with interval: 10 on a price field creates buckets [0,10), [10,20), etc. Date histograms use calendar-aware intervals (day, week, month, quarter) that handle variable-length periods and timezone offsets. The bucket key is the lower bound of the interval, and each bucket contains a document count plus any nested sub-aggregations.
Geographic aggregations group documents by spatial proximity. The geohash_grid aggregation divides the world into rectangular cells using geohash encoding at a configurable precision (1–12 characters). Higher precision means smaller cells and more buckets. geo_distance aggregation creates concentric rings around a center point, bucketing documents by distance. These aggregations enable map-based clustering and location-based faceting.
During query evaluation, the search engine scores documents and passes each matching document to the aggregation collector tree. The collector tree mirrors the nested aggregation structure: a root collector dispatches to bucket collectors, which dispatch to their metric sub-collectors. Each collector updates its state incrementally—no buffering of intermediate results.
// Zig-style aggregation collector interface
const Collector = struct {
collectFn: *const fn (self: *Collector, doc_id: u32, score: f32) void,
resultFn: *const fn (self: *Collector, allocator: Allocator) AggResult,
pub fn collect(self: *Collector, doc_id: u32, score: f32) void {
self.collectFn(self, doc_id, score);
}
pub fn result(self: *Collector, allocator: Allocator) AggResult {
return self.resultFn(self, allocator);
}
};
// Stats collector tracks running statistics
const StatsCollector = struct {
base: Collector,
field_reader: *DocValueReader,
count: u64 = 0,
sum: f64 = 0,
min_val: f64 = std.math.inf(f64),
max_val: f64 = -std.math.inf(f64),
// Welford's algorithm state for variance
mean: f64 = 0,
m2: f64 = 0,
pub fn init(field_reader: *DocValueReader) StatsCollector {
return .{
.base = .{
.collectFn = collectImpl,
.resultFn = resultImpl,
},
.field_reader = field_reader,
};
}
fn collectImpl(base: *Collector, doc_id: u32, _: f32) void {
const self = @fieldParentPtr(StatsCollector, "base", base);
const val = self.field_reader.getFloat64(doc_id);
self.count += 1;
self.sum += val;
if (val < self.min_val) self.min_val = val;
if (val > self.max_val) self.max_val = val;
// Welford's online algorithm
const delta = val - self.mean;
self.mean += delta / @as(f64, @floatFromInt(self.count));
const delta2 = val - self.mean;
self.m2 += delta * delta2;
}
fn resultImpl(base: *Collector, allocator: Allocator) AggResult {
const self = @fieldParentPtr(StatsCollector, "base", base);
const variance = if (self.count > 1) self.m2 / @as(f64, @floatFromInt(self.count - 1)) else 0;
return AggResult.stats(.{
.count = self.count,
.sum = self.sum,
.min = self.min_val,
.max = self.max_val,
.avg = if (self.count > 0) self.sum / @as(f64, @floatFromInt(self.count)) else 0,
.variance = variance,
.std_dev = @sqrt(variance),
});
}
};The histogram collector maps each document’s field value to a bucket key by integer division: bucket_key = floor(value / interval) * interval. Buckets are stored in a hash map keyed by bucket key. For date histograms, calendar math replaces simple division—the collector truncates timestamps to the start of their calendar period (day, week, month) using timezone-aware date functions.
// Go implementation of a numeric histogram aggregation
package aggregation
import "sort"
type HistogramAgg struct {
Field string
Interval float64
Buckets map[float64]*Bucket
}
type Bucket struct {
Key float64
DocCount int64
SubAggs []Aggregation
}
func NewHistogram(field string, interval float64) *HistogramAgg {
return &HistogramAgg{
Field: field,
Interval: interval,
Buckets: make(map[float64]*Bucket),
}
}
func (h *HistogramAgg) Collect(docID uint32, fieldValue float64) {
key := float64(int(fieldValue/h.Interval)) * h.Interval
bucket, ok := h.Buckets[key]
if !ok {
bucket = &Bucket{Key: key}
h.Buckets[key] = bucket
}
bucket.DocCount++
for _, sub := range bucket.SubAggs {
sub.Collect(docID, fieldValue)
}
}
func (h *HistogramAgg) SortedBuckets() []*Bucket {
result := make([]*Bucket, 0, len(h.Buckets))
for _, b := range h.Buckets {
result = append(result, b)
}
sort.Slice(result, func(i, j int) bool {
return result[i].Key < result[j].Key
})
return result
}The geohash grid aggregation converts each document’s lat/lon pair into a geohash string at the configured precision, then uses that string as the bucket key. Geohash encoding interleaves the bits of latitude and longitude into a base-32 string where each character adds roughly 2.5 bits of precision. At precision 5 (the common default), cells are approximately 4.9km × 4.9km.
package aggregation
// GeohashGridAgg buckets documents by geohash cell.
type GeohashGridAgg struct {
Precision int
Buckets map[string]*GeoBucket
}
type GeoBucket struct {
Geohash string
DocCount int64
CenterLat float64
CenterLon float64
}
func NewGeohashGrid(precision int) *GeohashGridAgg {
return &GeohashGridAgg{
Precision: precision,
Buckets: make(map[string]*GeoBucket),
}
}
func (g *GeohashGridAgg) Collect(docID uint32, lat, lon float64) {
hash := EncodeGeohash(lat, lon, g.Precision)
bucket, ok := g.Buckets[hash]
if !ok {
clat, clon := DecodeGeohash(hash)
bucket = &GeoBucket{Geohash: hash, CenterLat: clat, CenterLon: clon}
g.Buckets[hash] = bucket
}
bucket.DocCount++
}
// EncodeGeohash encodes a lat/lon pair into a geohash string.
func EncodeGeohash(lat, lon float64, precision int) string {
const base32 = "0123456789bcdefghjkmnpqrstuvwxyz"
var hash []byte
minLat, maxLat := -90.0, 90.0
minLon, maxLon := -180.0, 180.0
isLon := true
bits := 0
ch := 0
for len(hash) < precision {
if isLon {
mid := (minLon + maxLon) / 2
if lon >= mid {
ch = ch*2 + 1
minLon = mid
} else {
ch = ch * 2
maxLon = mid
}
} else {
mid := (minLat + maxLat) / 2
if lat >= mid {
ch = ch*2 + 1
minLat = mid
} else {
ch = ch * 2
maxLat = mid
}
}
isLon = !isLon
bits++
if bits == 5 {
hash = append(hash, base32[ch])
bits = 0
ch = 0
}
}
return string(hash)
}A search engine using terms aggregations can power faceted navigation. When a user searches for documents, the engine simultaneously computes term frequency counts for category, author, and tag fields. The frontend renders these as clickable filters: “Database (42)”, “Distributed Systems (28)”. Clicking a facet adds a filter to the query and re-aggregates, providing drill-down exploration with real-time counts.
Date histogram aggregations turn a search index into a lightweight time-series analytics engine. Log data indexed with timestamps can be aggregated by hour, day, or week to produce traffic charts, error rate graphs, or latency percentile trends. Nested stats aggregations inside date buckets compute per-period metrics without requiring a dedicated TSDB.
Geohash grid aggregations power map-based UIs where thousands of location results need to be clustered for display. At low zoom levels, a precision-3 geohash creates large cells (~156km) that cluster nearby results into a single marker with a count badge. As the user zooms in, the frontend requests higher precision (5, 6, 7) to split clusters into finer cells. This approach avoids transferring individual coordinates for millions of documents.
Aggregations enable dashboard-style UIs where every widget is a different aggregation on the same query scope. A product search page might show: a histogram of price distribution, a stats aggregation for average rating, a terms aggregation for top brands, and a geo_distance aggregation showing how many results are within 5km, 10km, 50km of the user. All computed in a single query pass.
When a terms aggregation only needs the top 10 buckets by document count, maintaining a min-heap of size 10 avoids sorting all buckets at the end. However, partial counts during collection mean the heap ordering is approximate. The standard approach uses a two-phase strategy: collect all buckets in a hash map during the query, then extract the top-N after collection is complete. For distributed search, each shard returns its local top-N and the coordinator merges them.
Exact cardinality (count of distinct values) requires storing every unique value. For high-cardinality fields (user IDs, IP addresses), this is too expensive. HyperLogLog estimates cardinality with ~2% error using only ~16KB of memory regardless of the actual cardinality. The aggregation hashes each field value and updates the HLL registers:
package aggregation
import "hash/fnv"
// HLLCardinalityAgg estimates distinct value count using HyperLogLog.
type HLLCardinalityAgg struct {
registers [16384]uint8 // 2^14 registers
precision uint // 14 bits
}
func NewHLLCardinality() *HLLCardinalityAgg {
return &HLLCardinalityAgg{precision: 14}
}
func (h *HLLCardinalityAgg) Add(value []byte) {
hasher := fnv.New64a()
hasher.Write(value)
hash := hasher.Sum64()
idx := hash >> (64 - h.precision) // top 14 bits = register index
remainder := (hash << h.precision) | (1 << (h.precision - 1)) // remaining bits
runLen := uint8(countTrailingZeros(remainder)) + 1
if runLen > h.registers[idx] {
h.registers[idx] = runLen
}
}
func (h *HLLCardinalityAgg) Estimate() float64 {
m := float64(len(h.registers))
sum := 0.0
zeros := 0
for _, reg := range h.registers {
sum += 1.0 / float64(uint64(1)< 0 {
estimate = m * logNatural(m/float64(zeros))
}
return estimate
}Percentile aggregations use the t-digest algorithm to maintain a compact, mergeable summary of a distribution. The t-digest clusters values into centroids weighted by count, keeping more centroids near the tails (where percentile accuracy matters most) and fewer in the middle. This allows accurate p50, p95, p99 estimates from a structure that consumes only a few KB regardless of how many values it has seen. The merge operation makes it ideal for distributed aggregation across shards.
For indexes with multiple segments, aggregation runs independently per segment and results are merged. Metric aggregations merge trivially (sum sums, min of mins). Bucket aggregations merge by combining buckets with the same key. This per-segment parallelism maps naturally to multi-threaded query execution, where each thread handles one segment.
A terms aggregation on a high-cardinality field (e.g., user IDs with 100 million unique values) creates 100 million buckets, consuming massive memory. Always set a size limit on terms aggregations and use cardinality estimation to check field cardinality before aggregating. For high-cardinality fields, consider using a composite aggregation that paginates through buckets.
In a distributed search system, each shard returns its local top-N term buckets. A term that ranks #11 on every shard (and thus is excluded from each shard’s response) might actually be the global #1. The standard mitigation is shard_size = size * 1.5 + 10, requesting more buckets from each shard than the final result needs. This reduces but does not eliminate the problem.
Date histograms default to UTC. A “daily” histogram in UTC splits a local business day across two buckets for users in UTC-8. Always specify the user’s timezone when creating date histogram aggregations. Also beware of DST transitions where a “day” is 23 or 25 hours.
Using too-high geohash precision creates millions of tiny buckets with 1–2 documents each, defeating the purpose of clustering. Too-low precision creates a few giant buckets that obscure geographic patterns. The precision should match the map zoom level: precision 3–4 for country-level views, 5–6 for city-level, 7+ for street-level.
Deeply nested aggregations (date histogram > terms > terms > stats) multiply the bucket count at each level. A monthly histogram (12 buckets) × 100 term values × 50 sub-term values = 60,000 bucket combinations, each with their own stats state. Limit nesting depth and bucket sizes, or use composite aggregations for deep drill-downs.