GGUF Quantized Tensor & Expert Decompression

Overview

GGUF (GPT-Generated Unified Format) is a binary file format designed by the llama.cpp project for storing quantized large language model weights. Unlike safetensors or PyTorch checkpoint files that store weights at full precision (FP32/FP16/BF16), GGUF files store tensors in aggressively quantized formats — from 2-bit (Q2_K) up to 8-bit (Q8_K) — that reduce model size by 4-16x while preserving most of the model’s generation quality. A Zig-based inference engine can implement a complete GGUF decoder that parses the file header, reads tensor metadata, and decompresses quantized blocks back to float32 for use in the inference engine.

Quantization in GGUF is not a simple per-element scaling operation. Each quantization type uses a block-based encoding scheme where a group of weights (typically 32 or 256 elements) shares one or more scale factors, zero-point offsets, and a set of packed low-bit integers representing the quantized values. The “K-quant” family (Q2_K through Q8_K) introduced in llama.cpp adds further complexity with super-blocks that contain nested sub-blocks, per-sub-block scales encoded at different bit widths, and minimum-value offsets that shift the quantization grid. Decompressing these formats requires bit-level manipulation: extracting 2-bit, 4-bit, or 6-bit fields from packed byte arrays, applying block-level and super-block-level scale factors, and adding minimum-value offsets — all in the correct order and with the correct arithmetic precision.

A second layer of complexity arises with Mixture-of-Experts (MoE) models such as Mixtral and DeepSeek. In these architectures, each transformer layer contains multiple “expert” sub-networks that share the same tensor name prefix but are distinguished by an expert index. GGUF stores all experts’ weights for a given layer as a single concatenated tensor, and the loader must slice this tensor along the correct axis to extract individual expert weight matrices. Getting the slicing wrong — using the wrong axis, the wrong stride, or the wrong offset — produces silent numerical errors that manifest as degraded generation quality rather than crashes, making this one of the hardest bugs to diagnose in a GGUF loader.

Core Concepts

How It Works

The GGUF loader operates in three stages: header parsing, tensor index construction, and on-demand decompression. The following code examples show the key mechanisms.

Parsing the GGUF Header and Tensor Index

The header parser reads the magic number, version, and metadata, then builds an index of tensor descriptors that maps tensor names to their byte offsets and quantization types. No tensor data is read at this stage — decompression happens lazily when a specific tensor is requested.

// gguf/parser.zig
const std = @import("std");
const Allocator = std.mem.Allocator;

pub const GGUFType = enum(u32) {
    f32 = 0,
    f16 = 1,
    q4_0 = 2,
    q4_1 = 3,
    q5_0 = 6,
    q5_1 = 7,
    q8_0 = 8,
    q8_1 = 9,
    q2_k = 10,
    q3_k = 11,
    q4_k = 12,
    q5_k = 13,
    q6_k = 14,
    q8_k = 15,
};

pub const TensorInfo = struct {
    name: []const u8,
    n_dims: u32,
    dims: [4]u64,
    type: GGUFType,
    offset: u64, // byte offset from start of data section
};

pub const GGUFHeader = struct {
    version: u32,
    n_tensors: u64,
    metadata: std.StringHashMap(MetaValue),
    tensors: []TensorInfo,
    data_offset: u64, // absolute file offset where tensor data begins
};

/// Parse the GGUF header and build the tensor index.
pub fn parseHeader(allocator: Allocator, reader: anytype) !GGUFHeader {
    // Read and validate magic number: "GGUF" as little-endian u32.
    const magic = try reader.readInt(u32, .little);
    if (magic != 0x46554747) return error.InvalidMagic; // "GGUF"

    const version = try reader.readInt(u32, .little);
    if (version < 2 or version > 3) return error.UnsupportedVersion;

    const n_tensors = try reader.readInt(u64, .little);
    const n_metadata = try reader.readInt(u64, .little);

    // Parse metadata key-value pairs.
    var metadata = std.StringHashMap(MetaValue).init(allocator);
    for (0..n_metadata) |_| {
        const key = try readString(allocator, reader);
        const val = try readMetaValue(allocator, reader);
        try metadata.put(key, val);
    }

    // Parse tensor info descriptors.
    const tensors = try allocator.alloc(TensorInfo, n_tensors);
    for (tensors) |*info| {
        info.name = try readString(allocator, reader);
        info.n_dims = try reader.readInt(u32, .little);
        info.dims = .{ 0, 0, 0, 0 };
        for (0..info.n_dims) |d| {
            info.dims[d] = try reader.readInt(u64, .little);
        }
        info.type = @enumFromInt(try reader.readInt(u32, .little));
        info.offset = try reader.readInt(u64, .little);
    }

    // Data section starts at the next 32-byte aligned boundary.
    const pos = try reader.context.getPos();
    const data_offset = (pos + 31) & ~@as(u64, 31);

    return .{
        .version = version,
        .n_tensors = n_tensors,
        .metadata = metadata,
        .tensors = tensors,
        .data_offset = data_offset,
    };
}

Q2_K Dequantization

The Q2_K decompressor is the most complex dequantization routine because it must extract 2-bit values from packed bytes and apply the two-level scale-and-minimum scheme. Each super-block produces 256 float32 values from 84 bytes of packed data.

// gguf/dequant.zig

/// Q2_K super-block layout (256 weights, 84 bytes):
///   - scales:  16 bytes (packed 4-bit sub-block scales, 16 sub-blocks)
///   - qs:      64 bytes (packed 2-bit quantized values, 256 values)
///   - d:       2 bytes  (FP16 master scale)
///   - dmin:    2 bytes  (FP16 master minimum)
const Q2K_BLOCK_SIZE: usize = 256;
const Q2K_BYTES: usize = 84;

pub fn dequantQ2K(src: []const u8, dst: []f32) void {
    const n_blocks = src.len / Q2K_BYTES;
    var dst_idx: usize = 0;

    for (0..n_blocks) |block_i| {
        const block = src[block_i * Q2K_BYTES ..][0..Q2K_BYTES];

        // Extract scales and minimums (packed 4-bit pairs in first 16 bytes).
        const scales_bytes = block[0..16];

        // Quantized 2-bit values are in bytes 16..80.
        const qs = block[16..80];

        // Master scale and minimum are the last 4 bytes (two FP16 values).
        const d = fp16ToF32(block[80..82]);
        const dmin = fp16ToF32(block[82..84]);

        // Process 16 sub-blocks of 16 weights each.
        for (0..16) |sb| {
            // Extract 4-bit scale and minimum for this sub-block.
            const scale_byte = scales_bytes[sb / 2];
            const sc: f32 = if (sb % 2 == 0)
                @floatFromInt(scale_byte & 0x0F)
            else
                @floatFromInt(scale_byte >> 4);

            const min_byte = scales_bytes[8 + sb / 2];
            const mn: f32 = if (sb % 2 == 0)
                @floatFromInt(min_byte & 0x0F)
            else
                @floatFromInt(min_byte >> 4);

            // Dequantize 16 weights in this sub-block.
            for (0..16) |elem| {
                const global_idx = sb * 16 + elem;
                const byte_idx = global_idx / 4;
                const bit_shift = @as(u3, @intCast((global_idx % 4) * 2));
                const quant: f32 = @floatFromInt((qs[byte_idx] >> bit_shift) & 0x03);
                dst[dst_idx] = d * sc * quant - dmin * mn;
                dst_idx += 1;
            }
        }
    }
}

/// Convert a 2-byte little-endian FP16 to f32.
fn fp16ToF32(bytes: *const [2]u8) f32 {
    const bits: u16 = @as(u16, bytes[1]) << 8 | @as(u16, bytes[0]);
    // Use Zig's built-in float conversion.
    const f16_val: f16 = @bitCast(bits);
    return @floatCast(f16_val);
}

MoE Expert Tensor Slicing

For Mixture-of-Experts models, the loader slices a concatenated expert tensor into individual expert matrices. The slicing must account for the quantization block size to ensure each slice begins at a block boundary.

// gguf/expert.zig
const std = @import("std");
const parser = @import("parser.zig");
const dequant = @import("dequant.zig");

/// Slice a packed MoE expert tensor and dequantize one expert's weights.
/// The tensor has shape [n_experts, rows, cols] stored as a flat
/// quantized byte array.  This function extracts expert_idx and
/// returns dequantized f32 values of shape [rows, cols].
pub fn sliceExpert(
    allocator: std.mem.Allocator,
    raw_bytes: []const u8,
    tensor_info: parser.TensorInfo,
    expert_idx: u64,
    n_experts: u64,
) ![]f32 {
    const rows = tensor_info.dims[1];
    const cols = tensor_info.dims[2];
    const elements_per_expert = rows * cols;

    // Compute the byte size per expert based on quantization type.
    const block_size = blockSizeFor(tensor_info.type);
    const bytes_per_element = bytesPerBlockFor(tensor_info.type);

    // Elements per expert must be divisible by block size.
    if (elements_per_expert % block_size != 0) {
        return error.ExpertSliceNotBlockAligned;
    }

    const blocks_per_expert = elements_per_expert / block_size;
    const bytes_per_expert = blocks_per_expert * bytes_per_element;

    const expert_start = expert_idx * bytes_per_expert;
    const expert_bytes = raw_bytes[expert_start..][0..bytes_per_expert];

    // Allocate output buffer and dequantize.
    const output = try allocator.alloc(f32, elements_per_expert);
    switch (tensor_info.type) {
        .q2_k => dequant.dequantQ2K(expert_bytes, output),
        .q4_0 => dequant.dequantQ4_0(expert_bytes, output),
        .q4_k => dequant.dequantQ4K(expert_bytes, output),
        .q6_k => dequant.dequantQ6K(expert_bytes, output),
        .q8_0 => dequant.dequantQ8_0(expert_bytes, output),
        .f16 => dequant.convertF16(expert_bytes, output),
        else => return error.UnsupportedQuantType,
    }

    return output;
}

fn blockSizeFor(qtype: parser.GGUFType) u64 {
    return switch (qtype) {
        .q4_0, .q4_1, .q5_0, .q5_1, .q8_0, .q8_1 => 32,
        .q2_k, .q3_k, .q4_k, .q5_k, .q6_k, .q8_k => 256,
        .f32 => 1,
        .f16 => 1,
    };
}

fn bytesPerBlockFor(qtype: parser.GGUFType) u64 {
    return switch (qtype) {
        .q4_0 => 18,   // 2 (scale) + 16 (4-bit packed)
        .q8_0 => 34,   // 2 (scale) + 32 (8-bit values)
        .q2_k => 84,   // super-block: scales + qs + d + dmin
        .q4_k => 144,  // super-block for Q4_K
        .q6_k => 210,  // super-block for Q6_K
        .q8_k => 292,  // super-block for Q8_K
        .f32 => 4,
        .f16 => 2,
        else => 0,
    };
}

Practical Applications

The GGUF loader powers on-device inference capability, allowing quantized models to run on machines with limited RAM. A 70B parameter model that requires 140 GB of RAM at FP16 precision fits in approximately 35 GB when quantized to Q4_K — within the reach of a workstation with 64 GB of RAM. The loader’s lazy decompression design means that only the tensors needed for the current inference pass are decompressed, and the decompressed float32 buffers are recycled between layers to minimize peak memory usage.

MoE expert slicing is critical for running models like Mixtral-8x7B, where each transformer layer contains 8 expert feed-forward networks. During inference, a routing mechanism selects the top-K experts (typically K=2) for each token, and only those experts’ weights need to be decompressed. The slicing API allows the inference engine to decompress only the selected experts per token, avoiding the cost of decompressing all 8 experts when only 2 are active — a 4x reduction in decompression work per layer pass.

The loader also supports model conversion workflows: reading a GGUF file in one quantization format, decompressing to float32, and re-quantizing to a different format. This is used when a model is available only in Q2_K (too lossy for some applications) and the user wants Q4_K quality. Rather than re-quantizing from the original FP16 weights (which may not be available), the conversion pipeline reads the Q2_K GGUF, decompresses to float32, and re-quantizes to Q4_K, accepting a small additional quantization error in exchange for avoiding a full-precision intermediate file.

Typical Integration Points

• An inference server’s model loading pipeline, which reads GGUF files from disk or S3 and populates the inference graph’s weight tensors.
• Model quantization tools that convert between GGUF quantization formats or from safetensors/PyTorch to GGUF.
• Memory-mapped inference on Apple Silicon, where the GGUF file is mmap’d and quantized blocks are decompressed directly from the mapped region using NEON SIMD intrinsics.
• Benchmarking harnesses that compare inference accuracy across quantization levels by loading the same model at Q2_K, Q4_K, Q6_K, and Q8_K and computing perplexity on a reference corpus.

Implementation Patterns

SIMD-Accelerated Dequantization

The inner loops of the dequantization routines process 16 or 32 elements at a time, which maps naturally to SIMD vector widths. Zig’s @Vector type expresses these operations in a platform-independent way: a @Vector(16, f32) holds 16 float32 values and maps to 4 SSE registers on x86 or 4 NEON registers on ARM. The compiler auto-vectorizes the scale-and-add operations, and explicit @shuffle operations handle the bit-unpacking for 2-bit and 4-bit quantized values.

Memory-Mapped File Access

Rather than reading the entire GGUF file into a heap buffer, the loader memory-maps the file and accesses tensor data via raw pointers into the mapped region. This lets the operating system manage paging — only the pages containing active tensors are resident in RAM, and the OS can evict pages for tensors that haven’t been accessed recently. For models larger than available RAM, this enables out-of-core inference where the OS pages tensor data in and out of disk as needed, albeit with significant performance overhead compared to in-memory access.

Block-Aligned Tensor Slicing

All tensor slicing operations (for MoE experts, layer extraction, or partial loading) must respect quantization block boundaries. The loader validates that the requested slice starts and ends on a block boundary before proceeding, returning a descriptive error if the model’s dimension sizes are not evenly divisible by the block size. This prevents a class of subtle bugs where a mis-aligned slice produces partially decompressed blocks whose scale factors belong to a different set of quantized values.

Quantization Type Dispatch via Comptime

Rather than using a runtime switch over quantization types in the hot path, the loader resolves the dequantization function once per tensor at load time and stores a function pointer. The dequantization function itself is specialized per quantization type at comptime, allowing the compiler to inline the type-specific constants (block size, bytes per block, bit widths) and eliminate branches from the inner loop. This is implemented as a comptime function that returns a fn([]const u8, []f32) void for a given GGUFType.

Common Pitfalls

Endianness in Scale Factor Decoding

GGUF stores all multi-byte values in little-endian format, including the FP16 scale factors in quantization blocks. On big-endian architectures (rare but not extinct), failing to byte-swap the scale factor before converting to float32 produces wildly incorrect dequantized values. The symptom is random-looking output tokens because every weight is scaled by a nonsensical factor. Always use explicit little-endian reads rather than casting byte pointers to *f16.

Off-by-One in 2-Bit Extraction

Extracting 2-bit values from packed bytes requires computing both the byte index (element_idx / 4) and the bit shift ((element_idx % 4) * 2). A common mistake is to use element_idx / 2 (which is correct for 4-bit but not 2-bit extraction) or to forget the * 2 on the shift, producing 1-bit values instead of 2-bit. The symptom is that half of the decompressed weights are zero, producing collapsed model outputs.

Incorrect Expert Stride for Non-Square Matrices

When slicing MoE expert tensors, the byte offset for expert k is k * rows * cols * bytes_per_element / block_size. A common mistake is to compute the offset as k * rows * bytes_per_row, which is incorrect when cols is not equal to the block-level stride. For K-quant types with 256-element super-blocks, the stride must account for the super-block structure — the byte boundary for the next expert may not be at cols bytes from the start of a row.

Version 2 vs. Version 3 String Encoding

GGUF version 2 stores string lengths as u32, while version 3 uses u64. Parsing a version 3 file with a version 2 string reader causes every string after the first to be misaligned by 4 bytes, corrupting all subsequent metadata and tensor info parsing. The result is typically a segfault or garbage tensor names. Always check the version field before selecting the string-reading code path.

Forgetting the 32-Byte Data Alignment

The GGUF data section starts at the next 32-byte aligned boundary after the header and tensor info sections. Tensor offsets in the tensor info array are relative to this aligned data start, not to the end of the tensor info section. Forgetting the alignment padding causes every tensor read to be shifted by up to 31 bytes, producing corrupted dequantization. The fix is to round up the current file position to the next multiple of 32 before reading any tensor data.

Further Reading