Model Quantization for Deployment (GPTQ, AWQ, GGUF, FP8)

Overview

Model quantization is the process of reducing the numerical precision of a neural network’s weights and activations — converting 32-bit or 16-bit floating-point numbers into lower-precision formats like 8-bit, 4-bit, or even 2-bit integers. This compression dramatically reduces memory footprint, increases inference throughput, and lowers compute costs, all while preserving most of the model’s output quality.

For AI engineers deploying large language models in production, quantization is often the single most impactful optimization technique. A 70B-parameter model that requires 140 GB of VRAM in FP16 can fit into a single 80 GB GPU when quantized to 4-bit precision. This difference determines whether you need one GPU or two, whether you can serve the model at all on consumer hardware, and what your per-token cost looks like at scale.

The modern quantization landscape includes four major approaches — GPTQ, AWQ, GGUF, and FP8 — each optimized for different hardware targets and quality-throughput tradeoffs. Understanding when to use each method is a core competency for LLMOps engineers.

How It Works

Quantization Fundamentals

At its core, quantization maps continuous floating-point values to a smaller set of discrete values. A FP16 weight that can represent ~65,000 distinct values gets mapped to an INT4 representation with only 16 distinct values. The key challenge is performing this mapping in a way that minimizes the impact on model output quality.

There are two broad categories of quantization. Post-training quantization (PTQ) takes a pre-trained model and converts its weights after training is complete. Quantization-aware training (QAT) incorporates quantization effects during training itself, generally producing higher quality results but requiring access to the training pipeline.

GPTQ (GPT Quantization)

GPTQ is a post-training quantization method that performs layer-wise quantization with second-order error correction. It processes each layer of the model independently, using a calibration dataset (typically 128-256 samples) to measure the impact of quantization and apply corrections that minimize the overall reconstruction error.

GPTQ is optimized for GPU inference and integrates with kernels like Marlin and ExLlama that deliver up to 3-5x speedups over FP16 on NVIDIA GPUs. It is the most widely supported 4-bit format in production inference engines like vLLM and TensorRT-LLM.

# Quantizing a model with GPTQ using AutoGPTQ
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,       # Quantization group size
    desc_act=False,       # Disable activation reordering for speed
    damp_percent=0.01     # Dampening factor for Hessian
)

model = AutoGPTQForCausalLM.from_pretrained(
    "meta-llama/Llama-3-70B",
    quantize_config=quantize_config
)
model.quantize(calibration_dataset)
model.save_quantized("./llama-3-70b-gptq-4bit")

AWQ (Activation-Aware Weight Quantization)

AWQ takes a different approach: instead of treating all weights equally, it identifies which weights are most important by analyzing activation patterns. The key insight is that a small fraction of weights (roughly 1%) carry disproportionate importance because they correspond to salient activation channels. AWQ preserves these critical weights at higher precision while aggressively quantizing the rest.

AWQ’s advantages include faster quantization time (no backpropagation required), better generalization across diverse tasks, and strong 4-bit quality — typically preserving ~95% of the original model’s performance. It works well with the vLLM and TensorRT-LLM inference engines.

GGUF (GPT-Generated Unified Format)

GGUF is a file format and quantization approach developed by the llama.cpp community, specifically optimized for CPU and Apple Silicon inference. Unlike GPTQ and AWQ which target GPU deployment, GGUF enables running large models on consumer hardware without a dedicated GPU.

GGUF supports a wide range of quantization levels using a k-quant scheme that groups weights and applies different precision levels to different groups based on their importance:

• Q2_K — 2-bit quantization, extreme compression (~3 bits/weight effective), significant quality loss
• Q4_K_M — 4-bit quantization, the sweet spot for most CPU deployments, good quality retention
• Q5_K_M — 5-bit quantization, nearly indistinguishable from FP16 for most tasks
• Q6_K — 6-bit quantization, near-lossless quality
• Q8_0 — 8-bit quantization, virtually lossless but larger files

The K in these names refers to k-quant mixed quantization, where attention and feed-forward layers may use different bit widths. The M suffix indicates “medium” quality within that bit level (vs. S for small/fast or L for large/quality).

FP8 (8-bit Floating Point)

FP8 is the newest entrant, supported natively by NVIDIA Hopper (H100) and Ada Lovelace (RTX 4090) architectures. Unlike integer quantization, FP8 maintains a floating-point representation, which better handles the wide dynamic range of transformer weights and activations.

FP8 on H100 GPUs delivers remarkable performance: 2x the throughput of FP16 with near-lossless quality. Benchmarks show Mistral 7B in FP8 achieves 33% faster output token generation and 8.5% lower latency compared to FP16. At batch size 128, an H100 serving FP8 Mistral 7B generates over 16,000 tokens per second. Memory footprint drops from 16 GB (FP16) to 7 GB (FP8), enabling 2x larger batch sizes on the same hardware.

Practical Applications

Choosing the Right Quantization Method

The choice of quantization method depends on your deployment target:

• GPU production serving (NVIDIA H100/A100): Use FP8 for maximum throughput with minimal quality loss. Falls back to AWQ or GPTQ at 4-bit if the model doesn’t fit in VRAM even at FP8.
• GPU serving with tight VRAM budgets: Use AWQ or GPTQ at 4-bit. AWQ generally provides better quality retention; GPTQ has broader kernel support and can be faster with optimized backends like Marlin.
• CPU/Apple Silicon deployment: Use GGUF with llama.cpp. Choose Q4_K_M for the best quality-to-size ratio, or Q5_K_M if you have the RAM.
• Edge devices and mobile: Use GGUF at Q2_K or Q3_K for aggressive compression, accepting quality tradeoffs.

Quality Benchmarks in Practice

Across standard benchmarks, quantization quality retention follows a predictable pattern. FP8 preserves 98-99% of original model quality and is effectively lossless for most tasks. AWQ at 4-bit retains approximately 95% quality. GGUF at Q4_K_M retains about 92% quality. GPTQ at 4-bit retains approximately 90% quality, though this improves significantly with larger group sizes (64 or 128).

Production Deployment Pipeline

A typical quantization pipeline involves: (1) selecting a calibration dataset representative of your production workload, (2) running the quantization process with your chosen method, (3) evaluating quality on your specific benchmarks and use cases, (4) load-testing the quantized model to measure throughput and latency improvements, and (5) deploying with an inference engine that supports optimized kernels for your chosen format.

# Serving a GPTQ model with vLLM
from vllm import LLM, SamplingParams

llm = LLM(
    model="TheBloke/Llama-3-70B-GPTQ",
    quantization="gptq",
    tensor_parallel_size=2,    # Split across 2 GPUs
    max_model_len=4096,
    gpu_memory_utilization=0.90
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Explain quantum computing"], sampling_params)

Key Takeaways

• Quantization reduces model precision (FP16 → INT4/FP8), cutting memory usage by 2-4x and increasing throughput proportionally.
• GPTQ uses layer-wise quantization with error correction — best for GPU inference with mature kernel support.
• AWQ identifies and preserves critical weights based on activation patterns — typically yields the best quality at 4-bit.
• GGUF is the standard format for CPU and Apple Silicon inference via llama.cpp, with flexible k-quant precision levels.
• FP8 delivers near-lossless quality with 2x throughput gains on NVIDIA Hopper/Ada GPUs — the preferred choice for high-end production serving.
• Quality retention at 4-bit ranges from 90-95% depending on method; FP8 retains 98-99%.
• Always benchmark quantized models on your specific use case — aggregate benchmarks don’t capture domain-specific quality impacts.
• The quantization method matters less than the bit width — the jump from FP16 to 4-bit is where most quality risk lives.