Edge deployment refers to running LLM inference directly on consumer hardware — laptops, desktops, smartphones, and embedded devices — rather than routing requests to cloud GPU clusters. This approach eliminates network latency, reduces per-query costs to zero after the initial hardware investment, and keeps sensitive data entirely on-device, which is critical for privacy-conscious applications.
The feasibility of edge LLM deployment has exploded since 2023, driven by two converging trends: aggressive model compression (quantization to 4-bit or lower) and optimized inference runtimes purpose-built for consumer hardware. Small models under 9B parameters, when properly quantized, now rival cloud-hosted models on specific tasks while running at interactive speeds on a MacBook or high-end phone.
Three frameworks dominate the edge inference landscape: llama.cpp for broad CPU/GPU compatibility, MLX for Apple Silicon optimization, and MLC-LLM for cross-platform compiled deployment including mobile. Understanding each framework’s architecture, performance characteristics, and ideal use cases is essential for choosing the right tool for your edge deployment.
llama.cpp is a C/C++ inference engine originally created by Georgi Gerganov to run Meta’s LLaMA models on consumer hardware without GPU dependencies. It has since evolved into the most widely-used local inference framework, supporting virtually every open-weight model architecture.
llama.cpp uses the GGUF (GPT-Generated Unified Format) model file format, which has become the de facto standard for distributable quantized models. When a new model is released, the first quantized versions on Hugging Face are almost always GGUF files.
Key architectural features of llama.cpp:
llama-server provides an OpenAI-compatible API for serving models locally.# Running a quantized model with llama.cpp
./llama-cli -m models/llama-3.1-8b-q4_k_m.gguf \
-p "Explain edge deployment in one paragraph" \
-n 256 \
--threads 8 \
--gpu-layers 35 # Offload layers to GPU if available
# Starting the OpenAI-compatible server
./llama-server -m models/llama-3.1-8b-q4_k_m.gguf \
--host 0.0.0.0 --port 8080 \
--ctx-size 4096 \
--parallel 4 # Handle 4 concurrent requestsOn an M2 MacBook Pro, llama.cpp achieves approximately 40-60 tokens/sec on a Q4-quantized 8B model. On desktop CPUs without GPU, expect 15-30 tokens/sec depending on core count and AVX support. With CUDA on an RTX 4090, throughput jumps to 100-150 tokens/sec.
MLX is Apple’s machine learning framework, designed from the ground up for Apple Silicon’s unified memory architecture. Unlike llama.cpp, which adapts to Apple hardware via the Metal backend, MLX is purpose-built to exploit the M-series chips’ unique capability of sharing memory between CPU and GPU without copying.
Apple Silicon’s unified memory architecture means the CPU and GPU share the same physical memory. MLX exploits this by performing operations on whichever processor is optimal without data transfer overhead — a fundamental advantage over discrete GPU architectures where data must be copied between CPU RAM and VRAM.
MLX’s mlx-lm package provides a high-level interface for loading and running LLMs:
# Install and run with mlx-lm
pip install mlx-lm
# Generate text
mlx_lm.generate \
--model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
--prompt "Explain edge deployment" \
--max-tokens 256
# Start an OpenAI-compatible server
mlx_lm.server --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bitPerformance benchmarks show MLX achieving approximately 230 tokens/sec on sustained generation tasks on M-series chips, with 5-7ms per-token latency. This is 30-50% faster than llama.cpp’s Metal backend on the same hardware for sustained throughput, though llama.cpp loads models significantly faster (~11 sec vs ~31 sec for MLX).
The trade-off: MLX only runs on Apple Silicon. If you need cross-platform compatibility, llama.cpp is the only option. If you’re building exclusively for the Apple ecosystem, MLX delivers measurably better performance.
MLC-LLM (Machine Learning Compilation for LLMs) takes a fundamentally different approach: it compiles models into optimized native code for each target platform using Apache TVM. This compilation step produces hardware-specific binaries that can run on iOS, Android, browsers (via WebGPU), and desktop platforms.
# Install MLC-LLM
pip install mlc-llm
# Compile a model for your target platform
mlc_llm compile ./models/llama-3.1-8b/ \
--quantization q4f16_1 \
--device metal # or cuda, vulkan, webgpu
# Run the compiled model
mlc_llm chat ./dist/llama-3.1-8b-q4f16_1/MLC-LLM achieves approximately 190 tokens/sec with ~13ms P99 latency in benchmarks — between llama.cpp and MLX on Apple Silicon, but with the advantage of truly cross-platform compiled binaries including mobile.
While not in the topic title, ExecuTorch deserves mention as the production-grade mobile inference runtime from Meta/PyTorch. It reached 1.0 GA in October 2025 with a remarkably small 50KB base footprint, supporting 12+ hardware backends and over 80% of popular edge LLMs out of the box. For production mobile apps shipping on the App Store or Google Play, ExecuTorch is increasingly the recommended choice.
Quantization is the enabling technology that makes edge deployment possible. Without it, even a 7B model requires 14 GB at FP16 — far more than most devices can accommodate. Key quantization formats for edge:
Healthcare, legal, and financial applications where data cannot leave the device benefit enormously from edge deployment. A medical office running a local 8B model for clinical note summarization keeps all patient data on-premises with zero cloud exposure. The model quality is sufficient for summarization tasks, and the latency is comparable to cloud APIs.
Field service applications, airline in-flight systems, and military deployments require AI capabilities without internet connectivity. A Q4-quantized 3B model running via llama.cpp on a rugged laptop provides a capable assistant that works in any environment.
Code completion and chat assistants running locally eliminate the cost and latency of cloud API calls. Tools like LM Studio, Ollama, and Jan wrap llama.cpp/MLX in user-friendly interfaces, enabling developers to run models like CodeLlama or DeepSeek-Coder locally with zero per-query cost.
# Framework selection guide
Use Case → Framework
────────────────────────────────────────────────
Cross-platform desktop/server → llama.cpp
Apple ecosystem (Mac/iOS) → MLX (or MLX + llama.cpp fallback)
Mobile app (iOS + Android) → MLC-LLM or ExecuTorch
Browser-based deployment → MLC-LLM (WebGPU)
Prototyping / quick testing → llama.cpp (via Ollama)
Production mobile shipping → ExecuTorch