Index Registry & Pluggable Index Architecture

Overview

An index registry is a centralized component that manages the lifecycle of multiple index implementations within a database engine. Rather than hardcoding each index type directly into the query planner or storage engine, a pluggable architecture decouples the index interface from its concrete implementations. This pattern allows a database system to support full-text search, dense vector similarity, sparse vector retrieval, and graph traversal through a single, unified abstraction layer.

In a typical database engine, this architecture is central to how different search modalities coexist. The registry acts as a factory and dispatcher: when a collection is configured with a particular index type, the registry instantiates the correct implementation and exposes it through a common interface. This means adding a new index type (say, a geo-spatial index) does not require modifying the query engine — it only requires registering a new implementation that satisfies the index interface.

The pluggable approach also enables enrichable interfaces, where an index can advertise additional capabilities beyond the base contract. For example, a dense vector index might also support filtering, or a full-text index might expose term-frequency statistics. By querying the registry for these enriched capabilities at runtime, the system can optimize query plans dynamically based on what each index actually supports.

Core Concepts

Registry Pattern

The registry pattern provides a single point of truth for all available index types. Each index implementation registers itself with the registry at startup, declaring its type identifier and a factory function. When the system needs to create an index for a collection field, it asks the registry to produce the correct implementation by type name.

Interface-Based Abstraction

All index implementations conform to a shared Go interface that defines core operations: inserting documents, deleting documents, and searching. This interface-based abstraction means the query planner never needs to know whether it is working with an HNSW graph, a BM25 inverted index, or a sparse vector store — it simply calls the interface methods.

Enrichable Interfaces

Beyond the base index contract, implementations can optionally satisfy additional interfaces that expose extra capabilities. The system uses Go type assertions at runtime to check whether an index supports filtering, aggregation, hybrid scoring, or other enriched behaviors. This avoids bloating the base interface while still allowing advanced optimizations.

How It Works

Index Type Registration

At application startup, each index package calls a registration function that adds its type and factory to the global registry. In a Go-based database engine, this follows Go’s init() convention — each index package has an init() function that registers itself. The registry stores a map from string identifiers like "fulltext", "dense_vector", "sparse_vector", and "graph" to their respective constructor functions.

Dispatch at Query Time

When a query arrives, the query planner examines the collection schema to determine which index types are configured for each field. It then asks the registry for the corresponding index instances. The registry either returns an already-loaded index or lazily initializes one via the factory function. This dispatch is transparent to the planner — it receives an object satisfying the Index interface regardless of the underlying implementation.

In a Zig-based database engine, the dispatch mechanism can be implemented with a tagged union and function pointer table, achieving the same polymorphism at the systems level. The Zig implementation uses comptime features to generate dispatch tables, eliminating virtual call overhead for known index types while preserving extensibility.

Enrichable Interface Discovery

After obtaining an index instance, the query planner can probe for enriched capabilities. For example, before executing a filtered vector search, the planner checks whether the vector index implements the Filterable interface. If it does, the planner can push filter predicates directly into the index scan rather than applying them as a post-filter. This capability negotiation happens at query planning time, not at registration time, so the same index instance can expose different capabilities depending on its current state or configuration.

In a well-designed database engine, enrichable interfaces are used extensively for cross-index capabilities. A full-text index that also implements VectorEnricher can participate in hybrid search pipelines, combining BM25 scores with vector similarity without requiring a separate fusion layer.

Practical Applications

Multi-Modal Search

The pluggable architecture makes multi-modal search straightforward. A single query can fan out across a full-text index and a dense vector index simultaneously, with results merged via a fusion algorithm (such as Reciprocal Rank Fusion). Because both indexes satisfy the same base interface, the query engine treats them uniformly when collecting and scoring results. Enrichable interfaces allow each index to contribute scoring metadata that the fusion layer can use for re-ranking.

Schema Evolution with New Index Types

When the database needs to support a new index type — for instance, a geo-spatial index for location-based queries — the development effort is isolated to implementing the Index interface and registering the new type. No changes are needed in the query planner, storage engine, or API layer. The registry pattern ensures that existing index types remain unaffected, and the new type is immediately available for any collection that declares it in its schema.

A/B Testing Index Implementations

The registry can hold multiple implementations for the same logical index type. For example, two different vector index algorithms (HNSW vs. IVF-PQ) can be registered under different names and assigned to different collections or even different shards of the same collection. This enables A/B testing of index performance, recall, and latency characteristics in production without modifying the application code. Metrics collected per-index-type feed back into decisions about which implementation to promote.

Implementation Patterns

Registry Pattern with Go Interfaces

The registry is typically implemented as a package-level variable holding a map[string]IndexFactory, protected by a sync.RWMutex for concurrent access. Each factory is a function that takes a configuration struct and returns an Index interface value. Registration happens at init time, and the registry is read-only during normal operation.

Factory Functions

Each index type provides a factory function that accepts configuration (dimensions, distance metric, analyzer settings, etc.) and returns a fully initialized index. The factory encapsulates all the complexity of setting up the underlying data structures — allocating memory, loading persisted state from disk, and configuring runtime parameters.

Capability Negotiation

Go’s type assertion mechanism (idx, ok := index.(Filterable)) serves as the capability negotiation protocol. The query planner maintains a list of optional interfaces it knows about and probes each index for support. This is more flexible than a capabilities bitmask because new capabilities can be added as new interfaces without modifying existing code.

Hot-Swapping Index Implementations

For zero-downtime upgrades, the registry supports atomic replacement of index instances. A new index is built in the background (either from scratch or by migrating data from the old index), and once ready, a compare-and-swap operation replaces the old instance in the registry. In-flight queries that hold a reference to the old index continue to completion, while new queries are routed to the new implementation. This pattern relies on Go’s garbage collector to clean up the old index once all references are released.

Code Examples

Index Registry Interface and Registration

The following example shows the core index interface, the registry structure, and how index types register themselves.

package indexregistry

import (
    "fmt"
    "sync"
)

// Index defines the base contract every index implementation must satisfy.
type Index interface {
    Add(id uint64, data []byte) error
    Delete(id uint64) error
    Search(query []byte, limit int) ([]Result, error)
    Flush() error
    Close() error
}

// Filterable is an enrichable interface for indexes that support
// predicate push-down during search.
type Filterable interface {
    SearchWithFilter(query []byte, filter Filter, limit int) ([]Result, error)
}

// Scorable is an enrichable interface for indexes that can expose
// raw relevance scores for fusion pipelines.
type Scorable interface {
    ScoreDetails(id uint64, query []byte) (ScoreBreakdown, error)
}

// IndexFactory is a constructor function that creates an Index
// from the provided configuration.
type IndexFactory func(cfg IndexConfig) (Index, error)

// Registry holds all registered index types and their factories.
type Registry struct {
    mu        sync.RWMutex
    factories map[string]IndexFactory
}

// Global registry instance.
var globalRegistry = &Registry{
    factories: make(map[string]IndexFactory),
}

// Register adds a new index type to the global registry.
// Typically called from an init() function in each index package.
func Register(typeName string, factory IndexFactory) {
    globalRegistry.mu.Lock()
    defer globalRegistry.mu.Unlock()
    if _, exists := globalRegistry.factories[typeName]; exists {
        panic(fmt.Sprintf("index type %q already registered", typeName))
    }
    globalRegistry.factories[typeName] = factory
}

// Create instantiates an index of the given type using its factory.
func Create(typeName string, cfg IndexConfig) (Index, error) {
    globalRegistry.mu.RLock()
    factory, ok := globalRegistry.factories[typeName]
    globalRegistry.mu.RUnlock()
    if !ok {
        return nil, fmt.Errorf("unknown index type: %s", typeName)
    }
    return factory(cfg)
}

Factory Implementation and Capability Negotiation

This example shows a concrete dense vector index registering itself with the registry, and how the query planner probes for enrichable interfaces at query time.

package densevector

import (
    "indexregistry"
)

func init() {
    // Register the dense vector index type at startup.
    indexregistry.Register("dense_vector", NewDenseVectorIndex)
}

// DenseVectorIndex implements HNSW-based approximate nearest neighbor search.
type DenseVectorIndex struct {
    dimensions int
    metric     DistanceMetric
    graph      *HNSWGraph
}

// NewDenseVectorIndex is the factory function for dense vector indexes.
func NewDenseVectorIndex(cfg indexregistry.IndexConfig) (indexregistry.Index, error) {
    dims := cfg.GetInt("dimensions", 128)
    metric := parseMetric(cfg.GetString("metric", "cosine"))

    graph, err := NewHNSWGraph(dims, metric)
    if err != nil {
        return nil, err
    }
    return &DenseVectorIndex{
        dimensions: dims,
        metric:     metric,
        graph:      graph,
    }, nil
}

func (d *DenseVectorIndex) Add(id uint64, data []byte) error {
    vec, err := decodeVector(data, d.dimensions)
    if err != nil {
        return err
    }
    return d.graph.Insert(id, vec)
}

func (d *DenseVectorIndex) Delete(id uint64) error {
    return d.graph.Remove(id)
}

func (d *DenseVectorIndex) Search(query []byte, limit int) ([]indexregistry.Result, error) {
    vec, err := decodeVector(query, d.dimensions)
    if err != nil {
        return nil, err
    }
    return d.graph.KNN(vec, limit)
}

// SearchWithFilter satisfies the Filterable enrichable interface,
// enabling predicate push-down into the HNSW traversal.
func (d *DenseVectorIndex) SearchWithFilter(
    query []byte,
    filter indexregistry.Filter,
    limit int,
) ([]indexregistry.Result, error) {
    vec, err := decodeVector(query, d.dimensions)
    if err != nil {
        return nil, err
    }
    return d.graph.FilteredKNN(vec, filter, limit)
}

func (d *DenseVectorIndex) Flush() error { return d.graph.Flush() }
func (d *DenseVectorIndex) Close() error { return d.graph.Close() }

// --- Query planner capability negotiation ---

func executeSearch(idx indexregistry.Index, query []byte, f *indexregistry.Filter, limit int) ([]indexregistry.Result, error) {
    // Probe for Filterable enrichable interface.
    if f != nil {
        if filterable, ok := idx.(indexregistry.Filterable); ok {
            // Push filter into the index scan for better performance.
            return filterable.SearchWithFilter(query, *f, limit)
        }
    }
    // Fall back to base search + post-filter.
    results, err := idx.Search(query, limit)
    if err != nil {
        return nil, err
    }
    if f != nil {
        results = applyPostFilter(results, *f)
    }
    return results, nil
}

Common Pitfalls

1. Bloating the base interface: Adding too many methods to the core Index interface forces every implementation to provide stubs for capabilities they do not support. Keep the base interface minimal and use enrichable interfaces for optional capabilities instead.
2. Registration order dependencies: If index types depend on other registered types during initialization (e.g., a hybrid index that wraps a full-text and vector index), circular or ordering issues can arise. Use lazy initialization or explicit dependency declarations to avoid init-time panics.
3. Ignoring thread safety in the registry: The registry is accessed concurrently by multiple goroutines during query execution. Failing to protect it with proper synchronization (e.g., sync.RWMutex) leads to data races. Read-heavy workloads benefit from RLock/RUnlock rather than exclusive locks.
4. Leaking resources on hot-swap: When replacing an index implementation at runtime, the old instance must be properly closed after all in-flight queries complete. Forgetting to call Close() on the replaced index causes memory leaks, open file descriptors, and corrupted flush state.
5. Over-relying on type assertions without fallbacks: Code that assumes an enrichable interface is always present will break when a new index implementation does not support it. Always provide a fallback path (e.g., post-filtering) when a type assertion returns false, and test with minimal index implementations that only satisfy the base interface.

Key Takeaways