Quantization

In short: a quantized row is a run of fixed-size blocks, 32 elements for the legacy types (Q4_0 to Q8_1) and 256 for the K-quants (Q2_K to Q8_K), each carrying its own f16 scale and, in some types, a minimum. The K-quants split a block into 8 or 16 sub-blocks with small integer scales of their own, scaled again by the block's f16 d. DequantizeRow turns any of them into f32 one block at a time, written to mirror the format rather than to be fast, and is checked against gguf-py. For decoding, VecDot never expands a weight to f32: the activations are quantized to int8 once (Q8_0 or Q8_K blocks that line up with the weight's), every product inside a block is an exact int32 sum, and only the block scales are float. That keeps the optimized kernel exactly comparable with its scalar reference, and on Apple Silicon each 16 multiply-adds are one sdot.

The concepts are on /ml-models: quantization and decoder-only LLMs. The module is four libraries, ggml_type, blocks (a header), dequantize and dot, with three tests and one benchmark (BUILD.bazel). rml's checklist records the module (progress.md L51) and the quantized GEMV on top of it (L65).

The type table

GgmlType numbers each element type exactly as ggml does, because that number is what a GGUF tensor record stores (ggml_type.h L18–40). A row is always whole blocks, so n elements of a type take n / elems · bytes bytes (L7–10). One private Layout switch holds the name, block elements and block bytes of every known id and returns nullopt for the rest (ggml_type.cc L17–61).

Every type the switch knows, with bits per weight worked out as block bytes · 8 / block elements:

Q4_K costs the same 4.5 bits as Q4_0 yet spends them better, which is the argument the module's header makes for K-quants (blocks.h L11–17); rml's memory notes use the same figures and put a llama.cpp Q4_K_M file, a mix of K-quant types, at about 4.9 (models.md L12–13, more on Models and memory).

Block layouts

blocks.h declares each block as a plain struct whose bytes match a GGUF file exactly, pinned by a static_assert on every size (blocks.h L111–120). Q8_0 and Q4_0 live in rml/core/dtype.h instead, because Tensor stores them as native dtypes (Tensors and dtypes, dtype.h L65–82).

Unpacking the packed fields

Every bit trick lives in small helpers in dequantize.cc, each with a comment stating the layout it reads. They are the readable definition of the formats:

Element 200 of a Q4_K block, traced through the helpers: it is in sub-block j = 200 / 32 = 6, so ScaleMin takes the pair-4-to-7 branch, scale = (s[10] & 0xF) | ((s[2] >> 6) << 4) and min = (s[10] >> 4) | ((s[6] >> 6) << 4). It is in chunk 200 / 64 = 3 at position 200 % 64 = 8, which is below 32, so q is the low nibble of qs[3 · 32 + 8] = qs[104]. Then x = d · scale · q − dmin · min (dequantize.cc L137–147).

Dequantization

The header calls these "the scalar definitions every optimized kernel is tested against": one block at a time, shaped like the format description, not tuned (dequantize.h L4–9).

Block data comes straight from a memory-mapped file, where a 34-byte or 210-byte block need not sit on its struct's alignment. So the generic DecodeBlocks copies each block into a local struct with memcpy before decoding it, and DecodeScalars does the same per element (L185–206). The dot products below make the opposite choice and require aligned pointers.

Quantization

rml quantizes only what it must produce itself: Q8_0 and Q4_0 rows, by round to nearest as ggml's reference does, and the activation formats the dot products need. It does not write K-quants; those come from GGUF files.

A worked example: Q4_0

Take a block whose first four values are −2.0, 1.3, 2.0 and 0.1 and whose other 28 are 0. The first value of largest magnitude is −2.0 (the later 2.0 ties and does not replace it, since the test is strict), so d = −2.0 / −8 = 0.25, exact in f16, and 1 / d = 4:

Mapping the extreme onto q = 0 uses all sixteen levels, −8 to +7 steps, so the value with the largest magnitude is exact; the price is that the opposite sign reaches only seven steps, and the 2.0 here comes back as 1.75 (dequantize.h L30–35). Packed, element j goes in the low nibble of qs[j] and element j + 16 in the high one, so with d's f16 bits 0x3400 stored little-endian the block begins 00 34 80 8D 8F 88: each high nibble is 8, element j + 16's zero.

Quantized dot products

VecDot(weight, w, x, n) is one weight row in its stored format against activations already quantized to int8, the inner loop of a decode-time matrix-vector product (dot.h L4–15). Each weight type is paired with an activation format whose blocks line up with its own, and within a block every product is an exact int32 sum:

DotActivationType returns that pairing and nullopt for every other type (dot.cc L285–297), and VecDot on an unpaired type is a programmer error (RML_CHECK, L339–356). The bsums are what make the K-quant minimum cheap: the sub-block's min term is min · Σa, and Σa over 32 activations is two of the stored 16-sums, one multiply instead of 32 (kernels.md L117–123).

Kernels and references

The kernels and the references share one float step per block (LegacyBlock, KBlockWithMin, KBlock, dot.cc L25–38), so they can differ only in how they reach the int32 sums. The design is one comment block (L59–68):

• Unpack, then dot. Each kernel unpacks a block into a small aligned int8 array in element order, a few shifts and masks per byte that the compiler vectorizes, then takes int8 × int8 → int32 dot products over 16 or 32 lanes.
• sdot where it exists. DotI8<N> uses vdotq_s32 when the compiler targets ARMv8.4 dotprod, which every Apple Silicon core has: one instruction multiplies 16 int8 pairs and adds each group of four into an int32 lane, so a 32-element block is two instructions and a horizontal add. Elsewhere it is a plain widening loop, which baseline SSE2 vectorizes through int16 and pmaddwd (L70–86). The sdot path is compiled and tested only on rml's macos-14 CI (kernels.md L130–136).
• References decode element by element. VecDotRef computes each q on its own, exactly as blocks.h describes it, and accumulates the same int32 sums; the K-quant reference sums min · a_i one element at a time instead of using bsums (L200–281, L358–375).
• Aligned input. Unlike DequantizeRow, the dot products cast the pointers to block arrays, so both must be aligned for their block types, as file mappings and vector buffers are; a debug check catches a misaligned pointer (dot.h L40–44, dot.cc L40–44).
• No overflow. rml's notes bound Q6_K's worst sub-block sum at 16 · 32 · 128 · 128, far inside int32 (kernels.md L120–123).

The benchmark

dot_benchmark times one 4096-element row against quantized activations, kernel and reference, with random block bytes (masked with 0x3B) so the result means nothing but the work is real (dot_benchmark.cc L13–41). rml's published numbers (kernels.md L138–147):

The unpack-then-dot shape is what turns Q4_K from the slowest reference into the fastest kernel. SSE2 has no int8 multiply, so every x86 product goes through int16; rml calls these numbers a floor, with the sdot path as the target. No Apple Silicon numbers for VecDot are published at this commit. The benchmark registers no reference for Q5_K, hence the dash.

Fixtures from gguf-py

tools/make_quant_fixtures.py checks rml against an independent implementation: gguf-py, MIT, from llama.cpp (L1–12). With NumPy's generator seeded at 0 it writes, for each of ten block types, two blocks of random bytes whose f16 scale fields (listed by byte offset in BLOCKS) are overwritten with finite values in ±0.05, and the f32 values gguf-py dequantizes them to; then, for Q8_0 and Q4_0, 96 normal values scaled by 2, with element 5 and elements 40–71 set to zero, and the bytes gguf-py quantizes them to (L23–35, L38–65). The result, testdata/gguf_quants.json, is about 55 KB and committed, so the C++ tests need no Python.

The script's comment on that range says an all-zero block has d = 0, but elements 40–71 straddle the second block (32–63) and the third (64–95), so neither is all zero and the d = 0 case is not in the fixture (L55–62). The C++ ZeroActivationsGiveZero test does quantize an all-zero row, through QuantizeActivations.

Who uses it

Rounding activations to int8 moves the toy models' logits by about 1e-2 against the float reference, which rml notes is what llama.cpp does too (kernels.md L149–155). The decoder that drives these products is on The decoder, the GGUF reader on Model files, and the threading and fusions of the products on LLM kernels.

Tests

The random weights in dot_test overwrite every f16 scale with a finite value in [0.01, 0.06) so the product is a number (dot_test.cc L28–55). The kernel test runs under ASan and UBSan as well (kernels.md L135–136).