Object-Storage-Native Search Architecture

Overview

Object-storage-native search systems use S3-style object storage as the durable source of truth for search data, while stateless or lightly stateful query nodes cache hot index data on NVMe SSD and in memory. This is different from a conventional search cluster where every shard owns local disks and replication is the primary durability mechanism. The durable index lives in object storage; compute nodes hydrate the pieces they need, serve queries, and can be replaced without moving ownership of the data.

The architecture is attractive for vector search, full-text search, and hybrid retrieval because search corpora are often much larger than their hot working sets. A namespace may contain millions or billions of vectors, but only a subset is queried repeatedly. Keeping the cold corpus in object storage lowers storage cost and simplifies failure recovery, while a cache hierarchy makes warm queries competitive with disk- or memory-resident engines.

The hard part is latency. Object storage has high request latency relative to local disk, so the index format and query planner must minimize roundtrips. Turbopuffer’s public architecture describes this explicitly: data is stored under namespace prefixes, writes append to an object-storage write-ahead log, indexing happens asynchronously, and cold queries are planned around a small number of large object-storage fetches before subsequent queries hit NVMe or memory cache.

Core Concepts

Object Storage as Durable Index State

The system treats object storage as the authoritative record for both write-ahead log entries and built index artifacts. Query and indexing workers can be stateless with respect to durability: if a node fails, another node can read the same namespace prefix and reconstruct the required state from durable objects.

Cache Hierarchy for Hot Namespaces

The durable tier is cheap but slow. A practical design layers NVMe SSD cache and memory cache above object storage. The first query against a cold namespace pays the hydration cost; later queries route to a node that already has the namespace’s index data cached. This makes cache locality a routing concern, not just a storage-engine concern.

Roundtrip-Aware Index Layout

Index data must be organized so a cold query can fetch coarse metadata, navigate to a small set of candidate partitions, and then perform a few large reads rather than hundreds of tiny reads. Centroid-based ANN indexes and block-oriented inverted indexes fit object storage better than pointer-heavy structures that require many dependent reads.

How It Works

Write Path: Append, Acknowledge, Then Index

Writes are first appended to a namespace-local write-ahead log in object storage. Once the log object is durable, the system can acknowledge the write. Background indexers then fold log entries into searchable vector, full-text, and metadata indexes. Until indexing catches up, queries can include a slower exhaustive scan over recent log records to preserve strong read-after-write behavior.

// Simplified write path for an object-storage-native search namespace.
func WriteBatch(ctx context.Context, ns Namespace, batch Batch) error {
    entry := WalEntry{
        Sequence: ns.NextSequence(),
        Records:  batch.Records,
    }

    key := fmt.Sprintf("%s/wal/%020d.json", ns.Prefix, entry.Sequence)
    if err := objectStore.Put(ctx, key, encode(entry)); err != nil {
        return err
    }

    // At this point the write is durable. Indexing can happen asynchronously.
    indexQueue.Enqueue(ns.ID, entry.Sequence)
    return nil
}

Read Path: Route for Cache Locality

A load balancer or query router should prefer a node that already has the target namespace cached. If no node is warm, any node can hydrate the needed metadata and index blocks from object storage. The router therefore tracks both ordinary health and search-specific locality: namespace ownership is soft, but cache affinity matters.

Cold Query Planning

Cold queries start by reading compact metadata: index version, partition directory, centroid summaries, block offsets, and filter statistics. The planner then chooses whether to spend another roundtrip to narrow the search or to fetch more candidate data in the current roundtrip. The right answer depends on object size, bandwidth, expected recall, and latency budget.

type ColdPlan struct {
    MetadataKeys []string
    CandidateBlocks []ObjectRange
    IncludeWalTail bool
}

func PlanColdVectorQuery(meta NamespaceMeta, q Vector, topK int) ColdPlan {
    centroids := selectNearestCentroids(meta.CentroidIndex, q, topK)
    blocks := make([]ObjectRange, 0, len(centroids))
    for _, c := range centroids {
        blocks = append(blocks, meta.ClusterObjectRange(c))
    }
    return ColdPlan{
        MetadataKeys: []string{meta.DirectoryKey},
        CandidateBlocks: coalesceRanges(blocks),
        IncludeWalTail: meta.HasUnindexedWrites(),
    }
}

Practical Applications

Large Multi-Tenant Vector Search

Managed vector search platforms often serve many tenants with very uneven activity. Object storage lets cold tenants remain inexpensive while active tenants benefit from cached namespaces. Compute can scale with query load rather than with total stored vector count.

Hybrid Search Over Archival Corpora

Support tickets, documentation, logs, media metadata, and compliance records can be too large to keep fully replicated on local SSD. An object-storage-native design can keep vector, BM25, and attribute-filter indexes close to the archival data while still supporting first-stage retrieval.

Elastic Search Compute

Because durable state is externalized, query workers can be replaced, autoscaled, or moved between regions with less shard-rebalancing work. The tradeoff is that new workers must warm caches before they deliver low-latency service for hot namespaces.

Implementation Patterns

Use Namespace Prefixes as Isolation Boundaries

Store each tenant or logical corpus under a stable object-storage prefix with subdirectories for WAL entries, index snapshots, metadata, and compaction manifests. Prefix-level organization simplifies listing, recovery, cache invalidation, retention, and per-namespace access controls.

s3://search-store/{namespace}/
  wal/
    00000000000000000001.json
    00000000000000000002.json
  index/
    manifest.json
    vector-centroids.bin
    vector-clusters-00017.bin
    bm25-postings-00042.bin
  metadata/
    filters.bin
    stats.json

Prefer Block and Partition Indexes

For vector search, centroid or partition-based layouts can fetch a few large cluster ranges. For full-text search, fixed-size posting blocks with term dictionaries and block-max metadata reduce key lookups. The common principle is to avoid graph traversals or posting scans that depend on many small remote reads.

Make Consistency a Query Option

Strong consistency may require reading recent WAL entries or checking object-storage metadata on every query. Some workloads can accept eventual consistency in exchange for lower warm latency. Expose this as an explicit query contract rather than silently returning stale results.

Performance Considerations

• Object-storage roundtrips: Latency is dominated by request count, not just bytes. Coalesce reads and design index files so each query needs a small bounded number of object operations.
• Cold versus warm latency: Cold queries hydrate metadata and candidate blocks from object storage; warm queries use SSD or memory cache. Publish and monitor both numbers because they describe different user experiences.
• Write latency: A durable object-storage WAL can make acknowledged writes slower than local-disk writes. Batch concurrent writes per namespace when possible, but make the batching delay visible in service-level objectives.
• Cache admission: Not every cold query deserves SSD residency. Use namespace size, repeated access, tenant priority, and predicted reuse to decide what enters cache.
• Index compaction: Asynchronous indexing creates manifests, segments, and old WAL tails. Compaction must reduce read amplification without creating large bursts of object-storage writes.
• Recall tradeoffs: Fetching fewer partitions lowers latency and cost but can reduce ANN recall. Tune candidate partitions against measured recall@k, not just query time.

Common Pitfalls

• Porting local-disk indexes unchanged: Pointer-heavy HNSW-like layouts or tiny posting files can perform poorly when every dependent read becomes an object-storage request.
• Ignoring cache routing: If the load balancer sends every query to a random node, warm namespaces repeatedly become cold. Cache-aware routing is part of the storage design.
• Hiding consistency costs: Strong read-after-write behavior may require reading an unindexed WAL tail. If this cost is invisible, teams will misinterpret warm-cache benchmarks.
• Letting WAL tails grow unbounded: Recent unindexed writes can be searched exhaustively for a while, but a stalled indexer turns that fallback into a query-latency incident.
• Measuring only average latency: A system can have excellent p50 warm latency and poor p95 cold latency. Track cold starts, cache evictions, and namespace hydration separately.