Large telemetry systems have an uncomfortable requirement: engineers need to search fresh logs, traces, counters, and job records immediately, but the data volume is too large for a conventional centralized indexing pipeline. If every source machine ships raw events to a central cluster and that cluster performs all parsing, projection, compression, and index construction, the ingestion layer becomes both expensive and slow. Queries then either wait for batch indexes to catch up or fall back to brute-force scans over massive raw streams.
Hyperscale asynchronous stream indexing solves this by splitting index construction across the places where data is produced and the services that expose it for querying. Source-side agents chunk incoming streams, extract indexed columns, compress records, and upload both data blocks and index fragments. A small strongly synchronized progress log records what has been committed. Background index builders merge fragments into larger global structures, while query engines see a relational access path that can be chosen or ignored by the optimizer.
Microsoft’s Helios system is the canonical production example. The VLDB paper describes a deployment that collects close to a quadrillion events and indexes roughly 16 trillion search keys per day across hundreds of thousands of machines. The core lesson is not the exact scale; it is the architecture. Fresh data can be queryable before every index level is fully compacted if the system separates durable data commit, asynchronous index advancement, and optimizer-visible access path selection.
A source-side agent runs near the data producer. It watches local streams, applies schema extraction or parsing rules, batches records into data chunks, and creates small index blocks for selected columns. This pushes expensive work toward machines that already see the data and can often spare some CPU. It also reduces central ingestion pressure because the backend receives structured chunks instead of raw firehose traffic that still needs full interpretation.
The system commits independently addressable data blocks to durable storage and separately creates index blocks mapping search keys to those data block locations. The data block is the source of truth; the index block is an acceleration structure. This distinction lets queries remain correct even while indexes lag, because the latest committed data can still be found through a bounded scan over recently committed blocks.
The progress log records commit progress for data and index layers. In Helios, it is the only strongly synchronized component. That design keeps consensus away from the bulk data path while still giving the system a durable ordering point for fault tolerance, watermark advancement, and query freshness. If an agent, ingestor, or index builder crashes, recovery can replay progress-log entries instead of guessing which blocks are safe to expose.
The index may be behind the newest data, but the queryable dataset does not have to be stale. Queries combine indexed access for compacted ranges with bounded scanning over committed tail ranges that have not yet entered the global index.
The ingestion path starts with a stream declaration that says where records come from, how they are parsed, which fields are indexed, where output is stored, and how records are chunked. The agent converts local stream updates into a batch, writes the data block to durable storage, produces index records for selected fields, and advances progress when the commit is durable.
CREATE STREAM ServiceLogs
FROM FileSystemSource("/var/log/service/*.json")
USING JsonExtractor()
(
event_time: datetime PRIMARY KEY,
tenant_id: string INDEX,
request_id: string INDEX,
worker_id: string INDEX,
message: string
)
OUTPUT TO "object://telemetry/service-logs"
PARTITION BY FORMAT(event_time, "yyyy/mm/dd/hh")
CHUNK EVERY 1 MINUTE;A practical implementation treats this declaration as a plan for two cooperating systems. The agent owns local extraction, batching, and fragment generation. The ingestion service owns durable placement, progress logging, and scheduling background index construction. Neither side should require the query engine to know how raw bytes were collected.
Rather than rebuilding a global tree for every arriving chunk, asynchronous builders merge leaf fragments into larger index nodes. Small or incomplete nodes can remain as orphans until enough adjacent material arrives to justify compaction. Internal levels advance through watermarks that describe how much of the stream has been absorbed at each level.
type DataBlock struct {
URI string
StartTime time.Time
EndTime time.Time
}
type IndexEntry struct {
Column string
Key string
Block string
}
func buildLeafIndex(block DataBlock, rows []Row, indexedColumns []string) []IndexEntry {
entries := make([]IndexEntry, 0)
for _, row := range rows {
for _, column := range indexedColumns {
if value, ok := row[column]; ok {
entries = append(entries, IndexEntry{
Column: column,
Key: normalize(value),
Block: block.URI,
})
}
}
}
sort.Slice(entries, func(i, j int) bool {
if entries[i].Column != entries[j].Column {
return entries[i].Column < entries[j].Column
}
return entries[i].Key < entries[j].Key
})
return entries
}Query engines should see the stream as a relation with optional secondary indexes. For a predicate such as tenant_id = 'acme', the optimizer can ask the index service for candidate blocks, read only those blocks, and apply the normal relational filter. If a predicate is not selective, or if the index is too far behind, the optimizer can prefer a scan. The index service is therefore a costed access path, not a separate query language.
During an outage, engineers often search by tenant, request ID, job ID, host, or failing component. These predicates are highly selective but span many machines. Source-side indexing lets the system expose searchable blocks quickly enough for incident response while keeping raw logs in durable storage for later forensic analysis.
Large compute fleets produce continuous telemetry about jobs, tasks, containers, network events, and resource usage. An asynchronous stream index can support both point lookups and aggregate exploration: find all records for a job, count failures by worker pool, or locate tenants whose traffic shifted during a deployment.
The same pattern applies outside hyperscale datacenters. Edge devices can summarize and index local sensor streams before uploading compact blocks. The central service receives queryable summaries and raw data pointers instead of every unprocessed event, which is useful when bandwidth, power, or cloud processing budget is constrained.
Treat data commit as the hard durability boundary. Once a block is durably written and recorded in the progress log, it belongs to the queryable dataset. Index construction can trail behind that boundary, and query planning can compensate by scanning the unindexed tail. This avoids coupling user-visible freshness to full global compaction.
func planLookup(index IndexClient, log ProgressLog, predicate Predicate, now CommitID) QueryPlan {
indexed, watermark := index.Lookup(predicate)
tail := log.BlocksAfter(watermark, now)
plan := QueryPlan{
CandidateBlocks: indexed.Blocks,
TailScanBlocks: tail,
}
if plan.EstimatedCost() > fullScanCost(predicate.TimeRange) {
return FullScan(predicate.TimeRange)
}
return plan
}Every index level should publish a watermark describing the newest commit it fully covers. Queries can use that watermark to bound the expensive part of the plan: scan only the committed blocks after the index watermark, not the entire dataset. If the tail grows too large, the system has a concrete operational signal that compaction is falling behind.
Consensus is expensive at high ingestion rates. The reusable pattern is to replicate the progress log and metadata transitions, not every byte of data or every index mutation. Durable object or file storage can hold large blocks, while the replicated log records enough metadata to recover ordering, ownership, and exposure decisions.