Running ML models locally—without calling external APIs—eliminates network latency, reduces operational costs, and keeps data private. ONNX (Open Neural Network Exchange) is the dominant interchange format for deploying trained models across runtimes: a model trained in PyTorch or TensorFlow is exported to .onnx format and executed by ONNX Runtime, a high-performance inference engine written in C++ with bindings for Go, Python, Rust, Java, and JavaScript.
For systems programming languages like Go and Zig, local inference requires bridging the gap between the C/C++ ONNX Runtime library and the host language. Go uses cgo or pre-built bindings to call the ONNX Runtime C API; Zig can call C libraries directly through its built-in C interop. The challenge is managing tensor memory, handling model loading, configuring execution providers (CPU, CUDA, CoreML, DirectML), and integrating inference into request-serving pipelines with acceptable latency.
As a concrete example, a production search system might use ONNX Runtime for local embedding generation and reranking. Instead of calling OpenAI or Cohere APIs for every document embedding, the system loads HuggingFace models exported to ONNX and runs inference locally. The gomlx library provides a higher-level Go interface for ML operations, while the raw ONNX Runtime C API gives maximum control over memory layout and execution providers. This hybrid approach enables offline operation, sub-millisecond inference latency for small models, and zero per-query API costs.
ONNX is a protobuf-based format that represents a computation graph. Each node in the graph is an operator (MatMul, Relu, Softmax, Attention, etc.) with typed inputs and outputs. The graph includes the model’s weights (stored as tensors in the protobuf or in external files for large models), metadata (input/output names, shapes, data types), and opset version (which set of operators the model uses). This self-contained format means any runtime that supports the declared opset can execute the model.
ONNX Runtime (ORT) loads the graph, applies optimizations (operator fusion, constant folding, layout transformations), and dispatches operators to execution providers. An execution provider (EP) is a backend that runs operators on specific hardware: the CPU EP uses optimized BLAS/LAPACK kernels, the CUDA EP runs operators on NVIDIA GPUs, CoreML EP delegates to Apple’s Neural Engine, and DirectML EP uses Windows GPU acceleration. ORT automatically partitions the graph, sending supported operators to the preferred EP and falling back to CPU for the rest.
Most embedding and reranking models are published on HuggingFace in PyTorch format. The optimum library exports them to ONNX with optimizations: operator fusion (combining multiple small ops into one), mixed precision (FP16 where accuracy permits), and quantization (INT8 for ~4x speedup with minimal accuracy loss). The exported .onnx file and tokenizer files are all that’s needed for inference—no Python runtime required.
Tensors in ONNX Runtime are contiguous arrays with a shape and data type. A batch of 8 text embeddings with 384 dimensions is a [8, 384] float32 tensor: 8 × 384 × 4 = 12,288 bytes of contiguous memory. When calling ORT from Go or Zig, you must allocate this memory, populate it with tokenized input, pass the pointer to ORT, and read results from the output tensor’s memory. Getting the memory layout wrong (row-major vs column-major, wrong data type, wrong shape) produces garbage results silently.
When creating an ORT session, you specify a priority-ordered list of execution providers. ORT tries each EP in order and falls back to CPU for unsupported operators. For example, requesting [CoreML, CPU] on a Mac runs compatible operators on the Neural Engine and the rest on CPU. This fallback is automatic and transparent—the same model file works on any hardware, just with different performance characteristics.
ONNX models expect pre-tokenized input: integer token IDs, attention masks, and segment IDs—not raw text. Tokenization (converting text to tokens using the model’s vocabulary) must happen outside ORT. In Python, the transformers library handles this. In Go/Zig, you need a separate tokenizer implementation (like the tokenizers crate via CGO or a pure Go implementation) that matches the model’s tokenizer exactly. A tokenizer mismatch produces valid-looking but incorrect embeddings.
INT8 quantization reduces model size by 4x and inference latency by 2–4x on CPU, with typically less than 1% accuracy degradation for embedding models. Dynamic quantization quantizes weights statically and activations at runtime; static quantization calibrates both with a representative dataset. For search/retrieval, the small accuracy loss is usually acceptable; for classification or generation tasks, benchmark carefully.
Using the onnxruntime-go bindings, load a model, create input tensors, and run inference:
package embedding
import (
"fmt"
ort "github.com/yalue/onnxruntime_go"
)
type EmbeddingModel struct {
session *ort.AdvancedSession
inputNames []string
outputNames []string
dimensions int
}
func NewEmbeddingModel(modelPath string, dimensions int) (*EmbeddingModel, error) {
// Initialize the ONNX Runtime shared library (call once per process)
ort.SetSharedLibraryPath("/usr/local/lib/libonnxruntime.so")
if err := ort.InitializeEnvironment(); err != nil {
return nil, fmt.Errorf("initializing ORT: %w", err)
}
// Configure session options
opts, err := ort.NewSessionOptions()
if err != nil {
return nil, err
}
defer opts.Destroy()
// Enable optimizations
opts.SetIntraOpNumThreads(4)
opts.SetInterOpNumThreads(2)
opts.SetGraphOptimizationLevel(ort.GraphOptLevelAll)
// Create the session with input/output specifications
inputNames := []string{"input_ids", "attention_mask"}
outputNames := []string{"embeddings"}
session, err := ort.NewAdvancedSession(modelPath, inputNames, outputNames, nil, nil, opts)
if err != nil {
return nil, fmt.Errorf("creating session for %s: %w", modelPath, err)
}
return &EmbeddingModel{
session: session,
inputNames: inputNames,
outputNames: outputNames,
dimensions: dimensions,
}, nil
}
func (m *EmbeddingModel) Embed(tokenIDs []int64, attentionMask []int64,
seqLen int) ([]float32, error) {
batchSize := int64(1)
shape := ort.NewShape(batchSize, int64(seqLen))
// Create input tensors from Go slices
inputIDsTensor, err := ort.NewTensor(shape, tokenIDs)
if err != nil {
return nil, fmt.Errorf("creating input_ids tensor: %w", err)
}
defer inputIDsTensor.Destroy()
maskTensor, err := ort.NewTensor(shape, attentionMask)
if err != nil {
return nil, fmt.Errorf("creating attention_mask tensor: %w", err)
}
defer maskTensor.Destroy()
// Create output tensor
outputShape := ort.NewShape(batchSize, int64(m.dimensions))
outputTensor, err := ort.NewEmptyTensor[float32](outputShape)
if err != nil {
return nil, fmt.Errorf("creating output tensor: %w", err)
}
defer outputTensor.Destroy()
// Run inference
err = m.session.Run(
[]ort.ArbitraryTensor{inputIDsTensor, maskTensor},
[]ort.ArbitraryTensor{outputTensor},
)
if err != nil {
return nil, fmt.Errorf("running inference: %w", err)
}
// Copy results from the output tensor
results := make([]float32, m.dimensions)
copy(results, outputTensor.GetData())
return results, nil
}
func (m *EmbeddingModel) Close() {
m.session.Destroy()
ort.DestroyEnvironment()
}Zig’s seamless C interop makes it straightforward to call the ONNX Runtime C API directly without wrapper libraries:
const std = @import("std");
const c = @cImport({
@cInclude("onnxruntime_c_api.h");
});
const OrtError = error{
CreateEnv,
CreateSession,
CreateTensor,
RunInference,
};
pub const EmbeddingModel = struct {
env: *c.OrtEnv,
session: *c.OrtSession,
allocator: *c.OrtAllocator,
api: *const c.OrtApi,
dimensions: usize,
pub fn init(model_path: [*:0]const u8, dimensions: usize) !EmbeddingModel {
const api = c.OrtGetApiBase().*.GetApi.?(c.ORT_API_VERSION);
var env: ?*c.OrtEnv = null;
var status = api.*.CreateEnv.?(c.ORT_LOGGING_LEVEL_WARNING, "inference-engine", &env);
if (status != null) {
api.*.ReleaseStatus.?(status);
return OrtError.CreateEnv;
}
// Create session with optimizations
var session_opts: ?*c.OrtSessionOptions = null;
_ = api.*.CreateSessionOptions.?(&session_opts);
_ = api.*.SetIntraOpNumThreads.?(session_opts, 4);
_ = api.*.SetSessionGraphOptimizationLevel.?(session_opts, c.ORT_ENABLE_ALL);
var session: ?*c.OrtSession = null;
status = api.*.CreateSession.?(env, model_path, session_opts, &session);
api.*.ReleaseSessionOptions.?(session_opts);
if (status != null) {
api.*.ReleaseStatus.?(status);
return OrtError.CreateSession;
}
var allocator: ?*c.OrtAllocator = null;
_ = api.*.GetAllocatorWithDefaultOptions.?(&allocator);
return EmbeddingModel{
.env = env.?,
.session = session.?,
.allocator = allocator.?,
.api = api,
.dimensions = dimensions,
};
}
pub fn embed(self: *const EmbeddingModel, token_ids: []const i64,
attention_mask: []const i64) ![]f32 {
const api = self.api;
const seq_len = token_ids.len;
// Create memory info for CPU allocation
var mem_info: ?*c.OrtMemoryInfo = null;
_ = api.*.CreateCpuMemoryInfo.?(c.OrtArenaAllocator,
c.OrtMemTypeDefault, &mem_info);
defer api.*.ReleaseMemoryInfo.?(mem_info);
// Create input tensors
const input_shape = [_]i64{ 1, @intCast(seq_len) };
var input_ids_tensor: ?*c.OrtValue = null;
_ = api.*.CreateTensorWithDataAsOrtValue.?(
mem_info,
@constCast(@ptrCast(token_ids.ptr)),
seq_len * @sizeOf(i64),
&input_shape, 2,
c.ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64,
&input_ids_tensor,
);
defer api.*.ReleaseValue.?(input_ids_tensor);
var mask_tensor: ?*c.OrtValue = null;
_ = api.*.CreateTensorWithDataAsOrtValue.?(
mem_info,
@constCast(@ptrCast(attention_mask.ptr)),
seq_len * @sizeOf(i64),
&input_shape, 2,
c.ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64,
&mask_tensor,
);
defer api.*.ReleaseValue.?(mask_tensor);
// Run inference
const input_names = [_][*:0]const u8{ "input_ids", "attention_mask" };
const output_names = [_][*:0]const u8{"embeddings"};
const inputs = [_]?*c.OrtValue{ input_ids_tensor, mask_tensor };
var outputs = [_]?*c.OrtValue{null};
const status = api.*.Run.?(
self.session, null,
&input_names, &inputs, 2,
&output_names, 1, &outputs,
);
if (status != null) {
api.*.ReleaseStatus.?(status);
return OrtError.RunInference;
}
defer api.*.ReleaseValue.?(outputs[0]);
// Extract output data
var output_data: ?*f32 = null;
_ = api.*.GetTensorMutableData.?(outputs[0], @ptrCast(&output_data));
const result = try std.heap.page_allocator.alloc(f32, self.dimensions);
@memcpy(result, output_data.?[0..self.dimensions]);
return result;
}
pub fn deinit(self: *EmbeddingModel) void {
self.api.*.ReleaseSession.?(self.session);
self.api.*.ReleaseEnv.?(self.env);
}
};For throughput, batch multiple inputs into a single inference call. This amortizes the overhead of GPU kernel launches and leverages SIMD parallelism on CPU:
func (m *EmbeddingModel) EmbedBatch(texts []string, tokenizer *Tokenizer) ([][]float32, error) {
// Tokenize all texts
maxLen := 512
batchSize := len(texts)
allTokenIDs := make([]int64, 0, batchSize*maxLen)
allMasks := make([]int64, 0, batchSize*maxLen)
actualLen := 0
for _, text := range texts {
tokens := tokenizer.Encode(text, maxLen)
if len(tokens.IDs) > actualLen {
actualLen = len(tokens.IDs)
}
allTokenIDs = append(allTokenIDs, tokens.IDs...)
allMasks = append(allMasks, tokens.AttentionMask...)
}
// Pad all sequences to the same length
paddedIDs := make([]int64, batchSize*actualLen)
paddedMasks := make([]int64, batchSize*actualLen)
for i := 0; i < batchSize; i++ {
srcStart := i * maxLen
dstStart := i * actualLen
copy(paddedIDs[dstStart:], allTokenIDs[srcStart:srcStart+actualLen])
copy(paddedMasks[dstStart:], allMasks[srcStart:srcStart+actualLen])
}
// Create batch tensors
shape := ort.NewShape(int64(batchSize), int64(actualLen))
inputTensor, _ := ort.NewTensor(shape, paddedIDs)
defer inputTensor.Destroy()
maskTensor, _ := ort.NewTensor(shape, paddedMasks)
defer maskTensor.Destroy()
outputShape := ort.NewShape(int64(batchSize), int64(m.dimensions))
outputTensor, _ := ort.NewEmptyTensor[float32](outputShape)
defer outputTensor.Destroy()
// Single inference call for the entire batch
err := m.session.Run(
[]ort.ArbitraryTensor{inputTensor, maskTensor},
[]ort.ArbitraryTensor{outputTensor},
)
if err != nil {
return nil, err
}
// Split output into individual embeddings
data := outputTensor.GetData()
results := make([][]float32, batchSize)
for i := 0; i < batchSize; i++ {
start := i * m.dimensions
results[i] = make([]float32, m.dimensions)
copy(results[i], data[start:start+m.dimensions])
}
return results, nil
}Creating an ORT session loads the model into memory, applies graph optimizations, and allocates internal buffers. This is expensive (100 ms–10 s depending on model size). Sessions are thread-safe for concurrent Run() calls, so create one session at startup and reuse it for all inference requests. Never create a new session per request. For multi-model setups, maintain a session pool keyed by model name.
A local search engine generates embeddings for document indexing and query encoding. Running a local model like all-MiniLM-L6-v2 (384 dimensions, 22M parameters, ~80 MB ONNX file) produces embeddings in 2–5 ms per text on CPU, versus 50–200 ms round-trip for an API call. For bulk indexing thousands of documents, local inference is orders of magnitude faster and cheaper. The model runs on the same machine as the index, eliminating network as a bottleneck.
Cross-encoder reranking models score query-document pairs for relevance. Running a local reranker like ms-marco-MiniLM-L-6-v2 scores 100 candidates in ~50 ms on CPU, enabling real-time reranking without API dependencies. The reranker takes the top-K results from the initial vector search and reorders them by a fine-grained relevance score.
Desktop and edge applications that must work without internet connectivity bundle ONNX models for local ML inference. For example, an offline-capable desktop search application can package the embedding model with the application binary, so all indexing and search happens locally. This is critical for enterprise deployments in air-gapped environments and for privacy-sensitive use cases where data cannot leave the device.
API-based embedding services charge per token or per request. At scale (millions of documents), the cost becomes significant. A local ONNX model running on a $500 GPU processes millions of embeddings per hour at near-zero marginal cost. The break-even point depends on volume, but for sustained workloads exceeding ~10,000 embeddings per day, local inference is almost always cheaper.
When supporting multiple models, use a registry pattern that loads models on demand and caches sessions:
type ModelRegistry struct {
mu sync.RWMutex
sessions map[string]*EmbeddingModel
basePath string
}
func NewModelRegistry(basePath string) *ModelRegistry {
return &ModelRegistry{
sessions: make(map[string]*EmbeddingModel),
basePath: basePath,
}
}
func (r *ModelRegistry) GetModel(name string) (*EmbeddingModel, error) {
// Fast path: check if already loaded
r.mu.RLock()
if model, ok := r.sessions[name]; ok {
r.mu.RUnlock()
return model, nil
}
r.mu.RUnlock()
// Slow path: load the model
r.mu.Lock()
defer r.mu.Unlock()
// Double-check after acquiring write lock
if model, ok := r.sessions[name]; ok {
return model, nil
}
modelPath := filepath.Join(r.basePath, name, "model.onnx")
config, err := loadModelConfig(filepath.Join(r.basePath, name, "config.json"))
if err != nil {
return nil, fmt.Errorf("loading config for %s: %w", name, err)
}
model, err := NewEmbeddingModel(modelPath, config.Dimensions)
if err != nil {
return nil, fmt.Errorf("loading model %s: %w", name, err)
}
r.sessions[name] = model
return model, nil
}
func (r *ModelRegistry) Close() {
r.mu.Lock()
defer r.mu.Unlock()
for _, model := range r.sessions {
model.Close()
}
}Bridging a HuggingFace tokenizer into Go using the tokenizers Rust library via CGO:
// Uses github.com/daulet/tokenizers (Go bindings for HuggingFace tokenizers)
type Tokenizer struct {
inner *tokenizers.Tokenizer
maxLen int
}
func NewTokenizer(tokenizerPath string, maxLen int) (*Tokenizer, error) {
tk, err := tokenizers.FromFile(tokenizerPath)
if err != nil {
return nil, fmt.Errorf("loading tokenizer from %s: %w", tokenizerPath, err)
}
return &Tokenizer{
inner: tk,
maxLen: maxLen,
}, nil
}
type TokenizerOutput struct {
IDs []int64
AttentionMask []int64
}
func (t *Tokenizer) Encode(text string) TokenizerOutput {
encoding := t.inner.Encode(text, true) // add special tokens
ids := encoding.IDs
if len(ids) > t.maxLen {
ids = ids[:t.maxLen]
}
// Convert uint32 IDs to int64 for ONNX Runtime
int64IDs := make([]int64, len(ids))
mask := make([]int64, len(ids))
for i, id := range ids {
int64IDs[i] = int64(id)
mask[i] = 1
}
return TokenizerOutput{
IDs: int64IDs,
AttentionMask: mask,
}
}The export pipeline converts HuggingFace models to optimized ONNX: (1) Export with optimum-cli export onnx --model sentence-transformers/all-MiniLM-L6-v2 ./model, (2) Optimize with onnxruntime.transformers.optimizer for operator fusion, (3) Optionally quantize to INT8 with onnxruntime.quantization. The resulting files (model.onnx, tokenizer.json, config.json) are versioned and distributed alongside the application binary.
intra_op_num_threads controls parallelism within a single operator (e.g., matrix multiplication). Set this to the number of physical cores (not hyperthreads) for CPU inference. inter_op_num_threads controls parallelism between independent operators. For serving workloads, lower inter-op threads prevent thread contention with the Go runtime.[batch, seq_len, dims]. You must apply mean pooling (averaging across the sequence dimension, masked by attention) to get a single embedding vector per input. Some ONNX exports include pooling; others don’t—check the model’s output shape.libonnxruntime with your application.Destroy() on tensors in a defer statement. Leaked tensors cause gradual memory growth that looks like a Go memory leak but won’t show in Go profiling tools.