A multi-backend ML inference engine provides a unified interface for running machine learning models across heterogeneous hardware and runtime environments. Rather than coupling your application to a single inference framework, this architecture abstracts backend selection behind a common contract, enabling the system to choose the best available runtime—ONNX Runtime for broad model compatibility, CoreML for Apple Silicon acceleration, XLA via GoMLX for high-throughput numerical computation, or pure Go for zero-dependency deployment—based on platform capabilities, model format, and performance characteristics.
This approach is critical for systems that embed ML scoring directly into the query path — for instance, a vector database that performs semantic reranking as part of search. When a search query triggers semantic reranking or embedding generation, the inference engine must resolve which backend can serve the loaded model with the lowest latency on the current hardware. On a macOS node with an M-series chip, CoreML may outperform ONNX Runtime for certain model architectures. On a Linux server with XLA-compatible accelerators, GoMLX provides the fastest path. The engine’s priority-based selection ensures optimal performance without requiring callers to understand backend specifics.
A key challenge in multi-backend inference is memory management across disparate runtime registries. Each backend maintains its own loaded model state, but total memory is shared. The cross-registry LRU eviction strategy described here treats all backends as participants in a single eviction pool, ensuring that the system stays within its memory budget regardless of which combination of backends is active. This design enables such a system to serve hundreds of models concurrently while maintaining predictable memory consumption.
Each registered backend declares a numeric priority and a CanServe(model) predicate. When a model inference request arrives, the engine iterates backends in priority order, selecting the first backend that can serve the given model format and is available on the current platform. This creates a deterministic fallback chain—for example, CoreML (priority 10) → ONNX Runtime (priority 20) → GoMLX (priority 30) → Pure Go (priority 100)—that gracefully degrades without caller intervention.
The ONNX Runtime backend provides the broadest model compatibility, supporting any model exported to the ONNX format from PyTorch, TensorFlow, scikit-learn, and other frameworks. It leverages the onnxruntime-go bindings to execute models with CPU, CUDA, or TensorRT execution providers. This backend serves as the primary workhorse for cross-platform inference and is typically the default fallback when hardware-specific backends are unavailable.
The CoreML backend targets Apple Silicon hardware, using Apple’s Core ML framework to exploit the Neural Engine, GPU, and CPU compute units available on M-series chips. Models are compiled to the .mlmodelc format at registration time, enabling sub-millisecond inference for small models. This backend is selected automatically on macOS/iOS platforms and provides significant latency improvements for embedding generation and reranking workloads compared to CPU-only ONNX execution.
The GoMLX backend wraps Google’s XLA (Accelerated Linear Algebra) compiler through a native Go interface. XLA compiles model graphs into optimized machine code for the target hardware, providing excellent throughput for batch inference. This backend excels at high-volume embedding generation where models can be expressed as GoMLX computation graphs, and it avoids the CGo overhead of ONNX Runtime by using XLA’s ahead-of-time compilation to produce native executables.
Each backend maintains its own internal model registry, but the inference engine overlays a unified LRU (Least Recently Used) eviction tracker across all registries. Every inference call updates a global access timestamp for the (backend, model) pair. When total memory across all backends exceeds the configured budget, the engine evicts the least recently used model from whichever backend holds it. This prevents any single backend from monopolizing memory and ensures fair resource sharing in mixed-workload scenarios.
The engine defines a common Backend interface that all inference backends must implement. A central registry holds all available backends, sorted by priority. This pattern allows new backends to be added without modifying the engine’s core selection logic.
// Backend defines the contract every inference backend must satisfy.
type Backend interface {
// Name returns a unique identifier for this backend (e.g., "onnxruntime", "coreml").
Name() string
// Priority returns the selection priority (lower = preferred).
Priority() int
// CanServe reports whether this backend can run the given model
// on the current platform with the available hardware.
CanServe(meta ModelMeta) bool
// LoadModel loads a model into the backend’s internal registry.
// Returns the memory consumed in bytes.
LoadModel(ctx context.Context, meta ModelMeta, data []byte) (uint64, error)
// Infer runs inference on a previously loaded model.
Infer(ctx context.Context, modelID string, input Tensor) (Tensor, error)
// UnloadModel removes a model from the backend, freeing its memory.
UnloadModel(modelID string) error
// MemoryUsage returns the total bytes consumed by loaded models.
MemoryUsage() uint64
}
// Registry holds all registered backends sorted by priority.
type Registry struct {
mu sync.RWMutex
backends []Backend // sorted by Priority() ascending
}
// Register adds a backend and re-sorts the priority list.
func (r *Registry) Register(b Backend) {
r.mu.Lock()
defer r.mu.Unlock()
r.backends = append(r.backends, b)
sort.Slice(r.backends, func(i, j int) bool {
return r.backends[i].Priority() < r.backends[j].Priority()
})
}When an inference request arrives, the engine walks the priority-sorted backend list and selects the first backend capable of serving the model. If a backend fails during inference, the engine can optionally fall back to the next eligible backend, providing resilience against transient runtime errors.
// SelectBackend returns the highest-priority backend that can serve the model.
func (r *Registry) SelectBackend(meta ModelMeta) (Backend, error) {
r.mu.RLock()
defer r.mu.RUnlock()
for _, b := range r.backends {
if b.CanServe(meta) {
return b, nil
}
}
return nil, fmt.Errorf("no backend available for model %q (format=%s)", meta.ID, meta.Format)
}
// InferWithFallback attempts inference on the preferred backend,
// falling back through the priority chain on transient errors.
func (e *Engine) InferWithFallback(ctx context.Context, modelID string, input Tensor) (Tensor, error) {
meta, ok := e.models[modelID]
if !ok {
return Tensor{}, fmt.Errorf("model %q not registered", modelID)
}
e.registry.mu.RLock()
backends := e.registry.backends
e.registry.mu.RUnlock()
var lastErr error
for _, b := range backends {
if !b.CanServe(meta) {
continue
}
result, err := b.Infer(ctx, modelID, input)
if err == nil {
e.lru.Touch(b.Name(), modelID)
return result, nil
}
lastErr = err
log.Printf("backend %s failed for model %s: %v; trying next", b.Name(), modelID, err)
}
return Tensor{}, fmt.Errorf("all backends failed for model %q: %w", modelID, lastErr)
}The cross-registry LRU cache maintains a doubly-linked list of (backend, modelID) pairs ordered by last access time. When memory pressure is detected, the engine evicts the tail entry—the least recently used model across all backends—and instructs the owning backend to unload it.
// LRUEntry tracks a loaded model’s last access across all backends.
type LRUEntry struct {
BackendName string
ModelID string
MemoryBytes uint64
LastAccess time.Time
}
// CrossRegistryLRU manages eviction across multiple backend registries.
type CrossRegistryLRU struct {
mu sync.Mutex
entries map[string]*list.Element // key: "backend:modelID"
evictList *list.List
maxMemory uint64
usedMemory uint64
}
// Touch updates the access time for a model, moving it to the front of the LRU list.
func (c *CrossRegistryLRU) Touch(backendName, modelID string) {
c.mu.Lock()
defer c.mu.Unlock()
key := backendName + ":" + modelID
if elem, ok := c.entries[key]; ok {
c.evictList.MoveToFront(elem)
elem.Value.(*LRUEntry).LastAccess = time.Now()
}
}
// Evict removes the least recently used model to free memory.
// It returns the backend name and model ID of the evicted entry.
func (c *CrossRegistryLRU) Evict() (string, string, error) {
c.mu.Lock()
defer c.mu.Unlock()
tail := c.evictList.Back()
if tail == nil {
return "", "", fmt.Errorf("no models loaded; nothing to evict")
}
entry := tail.Value.(*LRUEntry)
c.evictList.Remove(tail)
delete(c.entries, entry.BackendName+":"+entry.ModelID)
c.usedMemory -= entry.MemoryBytes
return entry.BackendName, entry.ModelID, nil
}
// EnsureMemory evicts models until the used memory is within budget.
func (e *Engine) EnsureMemory(needed uint64) error {
for e.lru.usedMemory+needed > e.lru.maxMemory {
backendName, modelID, err := e.lru.Evict()
if err != nil {
return fmt.Errorf("cannot free %d bytes: %w", needed, err)
}
backend, _ := e.registry.GetByName(backendName)
if err := backend.UnloadModel(modelID); err != nil {
log.Printf("warning: failed to unload model %s from %s: %v", modelID, backendName, err)
}
log.Printf("evicted model %s from backend %s to free memory", modelID, backendName)
}
return nil
}Backends are registered at application startup using build tags to conditionally include platform-specific implementations. On macOS, the CoreML backend is compiled in and registered at the highest priority. On Linux, the ONNX Runtime and XLA backends are preferred. The pure Go backend is always available as the ultimate fallback.
// RegisterDefaults adds all available backends for the current platform.
// Build tags control which backends are compiled into the binary.
func RegisterDefaults(reg *Registry) {
// Always available: pure Go fallback (priority 100)
reg.Register(NewPureGoBackend())
// Conditionally compiled via build tags:
// +build coreml → registers CoreML backend (priority 10)
// +build onnx → registers ONNX Runtime backend (priority 20)
// +build gomlx → registers GoMLX/XLA backend (priority 30)
registerPlatformBackends(reg)
}Cold inference requests can exhibit high tail latency due to lazy initialization in backend runtimes. The engine supports a warm-up phase where models are pre-loaded and a dummy inference pass is executed to trigger JIT compilation, memory allocation, and hardware-specific optimizations before the model begins serving real traffic.
// WarmUp pre-loads a model and runs a dummy inference pass.
func (e *Engine) WarmUp(ctx context.Context, meta ModelMeta, data []byte) error {
backend, err := e.registry.SelectBackend(meta)
if err != nil {
return err
}
memNeeded, err := backend.LoadModel(ctx, meta, data)
if err != nil {
return fmt.Errorf("warm-up load failed: %w", err)
}
if err := e.EnsureMemory(memNeeded); err != nil {
return fmt.Errorf("warm-up eviction failed: %w", err)
}
e.lru.Add(backend.Name(), meta.ID, memNeeded)
// Run a dummy inference to trigger JIT/compilation
dummy := NewZeroTensor(meta.InputShape)
if _, err := backend.Infer(ctx, meta.ID, dummy); err != nil {
log.Printf("warm-up inference warning for %s: %v", meta.ID, err)
}
return nil
}When a preferred backend becomes unavailable—for example, a GPU running out of memory or a CoreML compilation failure—the engine transparently retries on the next eligible backend. The InferWithFallback method shown above implements this pattern. Additionally, the engine emits structured metrics for each fallback event, enabling operators to detect when high-priority backends are chronically failing and require attention.
// backendHealthTracker monitors fallback frequency per backend.
type backendHealthTracker struct {
fallbackCounts map[string]*atomic.Int64
checkInterval time.Duration
}
// RecordFallback increments the fallback counter for a backend.
func (t *backendHealthTracker) RecordFallback(backendName string) {
if counter, ok := t.fallbackCounts[backendName]; ok {
counter.Add(1)
}
}
// CheckHealth returns backends exceeding the fallback threshold.
func (t *backendHealthTracker) CheckHealth(threshold int64) []string {
var unhealthy []string
for name, counter := range t.fallbackCounts {
if counter.Load() > threshold {
unhealthy = append(unhealthy, name)
}
}
return unhealthy
}When a vector database executes a semantic search query, the query planner identifies which ML models are needed for reranking candidate results. The inference engine loads the appropriate reranking model—typically a cross-encoder fine-tuned on domain-specific relevance data—into the best available backend. On Apple Silicon development nodes, CoreML provides sub-millisecond reranking per document pair. In production on Linux, ONNX Runtime with the TensorRT execution provider handles batch reranking across GPU-accelerated nodes. The pure Go backend enables reranking on edge nodes where no accelerator libraries are installed.
A database that generates vector embeddings at both index time and query time benefits greatly from this pattern. The multi-backend engine selects GoMLX for batch embedding generation during bulk indexing, where XLA’s graph compilation provides optimal throughput for processing millions of documents. For real-time query embedding, ONNX Runtime or CoreML is preferred due to lower per-request latency. The cross-registry LRU cache ensures that frequently used embedding models remain loaded while rarely accessed reranking models are evicted first, balancing memory across both workloads.
Hybrid search combines BM25 lexical scores with neural relevance signals. The inference engine runs lightweight feature extraction models that transform sparse lexical features into dense representations compatible with the vector index. Because these models are small, the pure Go backend can serve them with negligible overhead, reserving GPU memory and accelerator capacity for larger reranking and embedding models.
CanServe predicates and let the engine resolve the best backend at runtime. Hardcoded selection breaks portability and prevents graceful fallback.