ONNX (Open Neural Network Exchange) is an open-source interoperability standard for representing machine learning models. It defines a common file format and operator set that allows models trained in one framework — such as PyTorch, TensorFlow, or JAX — to be exported and deployed in a completely different runtime environment. The core promise is “train anywhere, deploy anywhere.”
For AI engineers, ONNX matters because production deployment rarely looks like the training environment. A model trained on PyTorch with CUDA GPUs may need to run on an ARM edge device, a Windows desktop with DirectML, or a cloud CPU fleet. ONNX provides the bridge, and ONNX Runtime (ORT) — Microsoft’s high-performance inference engine — is the most widely used execution backend for ONNX models.
However, ONNX was originally designed for traditional ML and computer vision workloads with static computation graphs. Modern LLMs with their dynamic shapes, autoregressive decoding loops, and massive KV caches expose fundamental tensions in the format’s design. Understanding both ONNX’s strengths and its limitations is critical for making informed deployment decisions.
An ONNX model is a directed acyclic graph (DAG) serialized as a Protocol Buffer (.onnx file). Each node in the graph represents an operation (a “operator”) from the ONNX operator specification, and edges represent tensors flowing between operations. The format captures:
ONNX operators are versioned in groups called “opsets.” A model declares which opset it targets (e.g., opset 21). This ensures that the runtime knows exactly which semantics each operator follows, preventing silent behavioral changes across versions.
Converting a model to ONNX involves tracing or scripting the model’s forward pass to capture it as a static graph. In PyTorch, this is done via torch.onnx.export():
import torch
import torch.onnx
model = MyModel()
model.eval()
# Create dummy input matching expected shape
dummy_input = torch.randn(1, 3, 224, 224)
# Export to ONNX format
torch.onnx.export(
model,
dummy_input,
"model.onnx",
opset_version=21,
input_names=["input"],
output_names=["output"],
dynamic_axes={
"input": {0: "batch_size"},
"output": {0: "batch_size"}
}
)The dynamic_axes parameter declares which dimensions can vary at runtime (e.g., batch size). This is straightforward for vision models but becomes complex for LLMs where sequence length, KV cache dimensions, and beam search width all vary dynamically.
ONNX Runtime is the primary inference engine for ONNX models. It accepts an .onnx file and optimizes it for the target hardware through a pipeline of graph transformations:
import onnxruntime as ort
# Create session with GPU execution provider
session = ort.InferenceSession(
"model.onnx",
providers=["CUDAExecutionProvider", "CPUExecutionProvider"]
)
# Run inference
result = session.run(
None,
{"input": input_array}
)While ONNX excels at traditional ML workloads (image classification, object detection, NLP embeddings), LLM inference introduces several challenges that push against the format’s design assumptions:
LLMs process variable-length sequences and grow their KV cache with each generated token. Some hardware execution providers (notably Qualcomm’s QNN-HTP for on-device inference) do not support dynamic shapes at all, requiring models to be compiled with fixed input dimensions. This forces padding to maximum sequence length, wasting compute, or using multiple compiled versions for different lengths.
Autoregressive LLM generation requires maintaining a KV (key-value) cache that grows with each decoded token. In ONNX, this cache must be explicitly modeled as additional model inputs and outputs, creating a feedback loop between inference steps:
Memory consumption scales linearly with context length. For Llama-2-7B in FP16 with batch size 1: 1K tokens requires ~500 MB of KV cache memory, 4K tokens requires ~2 GB, and 32K tokens requires ~16 GB. ONNX Runtime GenAI provides the GroupQueryAttention operator that supports past-present buffer sharing to avoid allocating separate memory for input and output KV caches.
ONNX fundamentally represents a static computation graph. Features that modern LLMs rely on — such as speculative decoding, dynamic routing in Mixture-of-Experts models, and variable-length beam search — require control flow that is difficult to express in ONNX’s operator set. While ONNX supports If and Loop operators, they are less optimized and not supported by all execution providers.
This is why dedicated LLM inference engines like vLLM, TensorRT-LLM, and llama.cpp have emerged as the preferred deployment path for large language models, while ONNX remains dominant for traditional ML model deployment at the edge and in heterogeneous environments.
ONNX’s primary value is enabling a single trained model to run across diverse hardware. A text embedding model trained in PyTorch can be exported to ONNX and deployed on Windows desktops via DirectML, iOS devices via CoreML, Android via NNAPI, or Qualcomm chipsets via QNN — all from the same .onnx file. This is invaluable for applications like on-device semantic search in mobile apps.
ONNX Runtime’s lightweight footprint and broad hardware support make it the go-to choice for edge deployment. Quantized ONNX models (INT8/INT4) can run on CPUs, NPUs, and mobile accelerators with minimal latency. AMD’s Quark toolchain and Qualcomm’s AI Engine Direct both consume ONNX as their primary model format for on-device AI.
For RAG pipelines, embedding models (sentence-transformers, E5, BGE) export cleanly to ONNX and benefit substantially from ORT’s graph optimizations. A typical embedding model sees 2-3x inference speedup after ONNX export with operator fusion and FP16 optimization, without any accuracy loss.
Models up to ~7B parameters can be effectively served via ONNX Runtime GenAI, especially when quantized to INT4. Microsoft’s Phi-3 family of small language models is specifically optimized for ONNX deployment, and the ONNX Runtime GenAI library handles the autoregressive loop, KV cache management, and sampling logic externally to the ONNX graph.
ONNX serves as a useful “checkpoint format” for validating model behavior across frameworks. Exporting to ONNX and comparing outputs ensures that framework-specific optimizations haven’t introduced numerical drift, which is critical for regulated industries requiring reproducible inference.