The decoder

In short: there is one model class and no per-family code. ModelConfig is a base recipe (pre-norm RMSNorm, RoPE, grouped-query attention, a gated FFN) plus a dozen switches (q/k/v bias, QK-norm, sliding window, sandwich norms, embedding scale, experts), read from config.json or GGUF metadata. WeightSource names every tensor by its GGUF name and maps Hugging Face names onto it, so both formats load through the same code, and matrices stay views into the mmapped file. DecoderLM::Forward runs any number of tokens at positions cache.size() onward: embed, then per layer norm → q/k/v → RoPE → store in the KV cache → one-pass online-softmax attention → output projection added into the residual, then norm → FFN (or top-k experts) added into the residual. After a prefill, decoding one token at a time allocates nothing, and each family is pinned by golden logits from a float64 reference on toy models in both formats, plus real-checkpoint goldens.

The concepts are on /ml-models: decoder-only LLMs, attention, the KV cache, positional encoding and mixture of experts. The code is in rml/llm: config, weight_source, decoder, attention, rope, kv_cache and kernels. The files it reads are on Model files, the weight formats it multiplies on Quantization, the token ids it takes on Tokenizers, and what is done with its logits on Generation.

The model config

config.h states the design in its header: every supported model is one recipe with switches, and the architectures differ only in which switches they flip (config.h L4–20). ModelConfig carries the sizes and those switches (L59–111):

From config.json

ParseHfConfig accepts eight model_types: llama, mistral, mixtral, qwen2, qwen3, qwen3_moe, gemma3_text and olmoe (config.cc L23–24), and returns Unsupported for anything else (L292–367). Along the way it:

• takes the activation from hidden_activation or hidden_act (SiLU or gelu_pytorch_tanh) and rejects mlp_bias;
• defaults num_key_value_heads to the head count and head_dim to hidden / heads, and rms_norm_eps to 1e-5 for Mixtral and OLMoE, 1e-6 otherwise, the defaults of transformers' config classes (L326–329);
• reads RoPE from rope_parameters (preferring its full_attention set) or rope_scaling (L83–113);
• derives the switches from the architecture: Gemma ties its head by default; Qwen2 always has q/k/v bias; Qwen3, Qwen3-MoE, Gemma and OLMoE have QK-norm, OLMoE's spanning every head (L339–345);
• rejects OLMo's clip_qkv and Qwen's use_sliding_window, and reads sliding_window only for non-Qwen models, since Qwen configs carry one they do not use (L346–356);
• reads the expert keys per family (ParseMoe, L185–206) and Gemma's extras (ParseGemma, L115–162), then the BOS and EOS ids, and finally ValidateConfig.

ValidateConfig checks every size is positive, the heads split evenly into kv groups, head_dim is even (RoPE rotates pairs), the epsilon, theta, scaling factor and scales are positive, sliding_layers has one flag per layer, and experts per token lie in [1, num_experts] (L257–290).

From GGUF metadata

ModelConfigFromGguf reads general.architecture and then the <arch>.* keys (L379–493). GGUF names differ: llama becomes mixtral when it has an expert_count, qwen3moe becomes qwen3_moe and gemma3 becomes gemma3_text. rml's fixtures write Mistral as llama with an attention.sliding_window key (make_llm_fixtures.py L196–197), so it loads as a Llama with a window. Partial rotary embeddings are rejected. The switches GGUF has no key for are read off the tensor table instead (L467–473): a missing output.weight means a tied head, a blk.0.attn_q.bias means q/k/v bias, a blk.0.attn_q_norm.weight means QK-norm (whole when its length is not head_dim), and a rope_freqs.weight tensor means Llama 3.1's precomputed frequency factors. The vocabulary size falls back to the embedding's row count.

Expert weights are renormalized for every MoE architecture but OLMoE unless the file has expert_weights_norm, following llama.cpp (L421–436). Gemma 3's GGUF keys get their own reader: an explicit gemma3.attention.scale, or llama.cpp's default (the 27B model, 62 blocks, scales by hidden / heads, the rest by head_dim); the local RoPE base; and the sliding pattern as a period or one flag per layer (L208–253).

ParameterCount counts attention, FFN or experts plus router, two or four norms per layer, the final norm and the embedding, the head only when untied (L495–506); rml's memory tables are built from that count (Models and memory).

Weights: WeightSource

A model's tensors are looked up by their GGUF names whichever file holds them; a Hugging Face checkpoint is read through the matching name, and matrices are views into the mapped files while only norm weights and biases are copied out as f32 (weight_source.h L4–11).

weight_source.cc L33–88. Note how Gemma reuses the name post_attention_layernorm for a different norm: in Llama it is the norm before the FFN, in Gemma the sandwich norm after attention, so Gemma's table is checked first.

Matrix(name, rows, cols) fetches the raw bytes, checks the shape, and wraps them in a WeightMatrix: a type, a row and column count, and a span of the mapping (L134–167; kernels.h L22–37). GGUF tensors keep their stored type (F32, F16, BF16, Q8_0, Q4_K, …); safetensors tensors must be f32, f16 or bf16. MakeWeightMatrix checks the columns are whole blocks and the byte count is exact, and dequantizes one block as a probe so integer types, which have block layouts but no meaning as weights, are refused (kernels.cc L43–64).

Experts come two ways (L169–197): GGUF stacks each layer's experts into one [experts, rows, cols] tensor (ffn_gate_exps, ffn_up_exps, ffn_down_exps) and expert e is the e-th slice of its bytes; Hugging Face stores one matrix per expert, block_sparse_moe.experts.E.w1/w3/w2 in Mixtral and mlp.experts.E.gate_proj/up_proj/down_proj in Qwen3-MoE and OLMoE. Vector converts a norm or bias to f32, and adds 1 to a Hugging Face Gemma norm weight, because Gemma's RMSNorm scales by (1 + w) and GGUF converters already store 1 + w (L199–217). One RmsNorm then serves every family. A known gap: for a safetensors tensor that is not aligned, Raw keeps a span of a copy that is freed when it returns; see Model files.

DecoderLM::LoadLayers asks for every tensor the config implies (q/k/v biases only with attention_bias, q/k norms only with qk_norm, sandwich norms only for Gemma, a router and num_experts × 3 expert matrices instead of a dense FFN for MoE), so a missing or misshapen tensor fails at load with its file name and shapes, not mid-forward (decoder.cc L100–159). The embedding is a WeightMatrix too; Forward dequantizes just the rows of the tokens it sees (L226–233).

One forward pass

Each block is pre-norm: x += o_proj(attention(rope(q(norm(x))), rope(k(norm(x))), v(norm(x)))) then x += ffn(norm(x)), and Gemma normalizes each branch's output too before the add (decoder.h L10–25). Forward(tokens, cache, logits, all_logits) runs n tokens at positions cache.size() onward; the same call is a prefill (n = prompt length) or one decode step (n = 1) (decoder.cc L204–273).

Step by step:

1. Checks. An empty input, an id outside the vocabulary, a logits span of the wrong size or a cache made for another model is InvalidArgument; a full cache is OutOfRange from cache.Extend(n), and in every case nothing has changed yet (L206–223).
2. Scratch. Reserve(n) grows each Workspace buffer to the largest call so far: x, normed, q, k, v, the attention output, Gemma's branch buffer, the gate, the router scores, the cos and sin tables, and the int8 input scratch sized for the largest matrix (L165–197). After a prefill, a one-token step never grows anything.
3. Embed. Each token's embedding row is dequantized into x, times embedding_scale (√hidden for Gemma); then the cos and sin of every rotary pair are computed once per position, for the global RoPE and, when there is one, the local one (L226–242).
4. Attention half. RmsNorm each row into normed, then Residual runs Attention into the stream (L289–325): MatMulMany computes q, k and v; biases are added; QK-norm normalizes each head (or each whole row); RoPE rotates q and k in place; k and v go to the cache; Attend writes the attention output; and the output projection is a MatMul with accumulate = true, so it adds straight into x.
5. FFN half. RmsNorm again, then DenseFfn (MatMulGlu writes act(gate) × up, the down projection accumulates into x) or MoeFfn (L327–331).
6. Sandwich norms. With sandwich_norms, Residual zeroes a branch buffer, runs the block into it, normalizes each row with the post norm and only then adds it to x (L275–287).
7. Logits. Only the rows that need logits (the last, or all with all_logits) get the final norm and the output product (L266–272).

The kernels underneath

The arithmetic lives in kernels.h, written as plain loops over f32 activations and weights in their stored format; a quantized row is expanded a few blocks at a time, never as a whole matrix (kernels.h L4–10). Everything here is parallel over the output rows of the weight in grains of 16 (kernels.cc L149–156) and never allocates.

The three fusions (shared input, gated FFN, residual add as accumulate) and their measurements are on Kernels; the int8 dot products on Quantization.

Attention and the KV cache

The paged cache

KvCache keeps every earlier token's keys and values, in pages of page_size tokens (NewCache uses 16), each page holding that span for every layer. A page table maps a position to its page, and all pages come out of one arena reserved up front, so nothing is allocated while decoding and untouched pages are never made resident (kv_cache.h L4–19). The arena is calloced because large zeroed allocations come straight from mmap (kv_cache.cc L20–62).

A row is one position's keys (or values) for one layer, all kv heads: kv_size entries. Inside the arena rows are ordered page, then layer, then key/value, then slot: row = ((page · num_layers + layer) · 2 + which) · page_size + position mod page_size (kv_cache.h L102–109), so one layer's keys for a page are page_size contiguous rows.

An int8 cache stores each group of channels as round(x / s) with one f32 scale s = max|x| / 127. The group is the largest power of two dividing head_dim, at most 32, so groups never straddle two heads, and a row is its scales first (keeping them 4-byte aligned) then its values padded to a multiple of 4 (L37–46, L84–98).

Worked example: Llama 3.1 8B. 32 layers, 8 kv heads of 128, so kv_size = 1024.

Attend

Query i at position start + i attends to every cached position j from the window start (0, or pos − window + 1) to itself, through the kv head its query head shares with its group, and Attend does it in one pass of online softmax: a running maximum and normalizer per query head, the partial output rescaled when the maximum grows, no score array ever materialized (attention.h L4–16).

Attend picks AttendGroup<float>, <Float16> or <int8_t> by the cache's dtype and runs n × num_kv_heads tasks on the pool with grain 1 (attention.cc L141–166). Inside a task (L67–129) keys are taken 32 at a time, each key row widened to f32 into a stack buffer as it is read (HeadRow, L42–65) and dotted against every query in the group with four partial sums (L24–37). Per-task state lives on the stack, which is why the group is limited to 16 query heads and head_dim to 512 (attention.h L34–37). AttendRef is the definition: every score, a softmax, a weighted sum (L168–213).

For Llama 3.1 8B a decode step is 8 tasks per layer, each serving 4 query heads. rml's own measurement of one decode step's attention at that shape, on one x86 core of the development container: 1.3 ms at 512 tokens and 25 ms at 4096 with an f32 cache, against 3.3 and 64 ms for AttendRef; 32 ms with f16 (no F16C on that SSE2 build) and 19 ms with int8 at 4096 (kernels.md L170–194, from attention_benchmark).

Rotary embeddings

Pair i of a head's dimensions turns at inv_freq[i] = θ^{−2i / head_dim} radians per position, so q · k depends only on the distance between two tokens; the scaling schemes stretch the slow pairs so a model trained on a short context reads positions beyond it (rope.h L4–11). Rope::Create computes the frequencies and applies the scaling once (rope.cc L73–112); Angles gives each pair's cos and sin at a position, times YaRN's attention factor (L114–121); Apply rotates heads in place (L123–140).

Worked example: Llama 3.1 8B (factor 8, low and high frequency factors 1 and 4, original context 8192; models.md L26–28). Pairs whose wavelength 2π / inv_freq is under 8192 / 4 = 2048 positions keep their frequency, those over 8192 / 1 = 8192 turn eight times slower, and between them smooth = (8192 / wavelength − 1) / 3 blends the two, so a pair with a wavelength of 4096 positions gets (1 − 1/3) of the slowed frequency plus 1/3 of the original.

Which dimensions pair. Hugging Face's rotate_half pairs dimension i with i + head_dim/2 (kHalf); llama.cpp's converter permutes Llama-family q and k rows so that pairs are adjacent, (2i, 2i + 1) (kInterleaved) (rope.h L22–29). rml does not undo the permutation: it rotates in whichever order the file uses. FromHf always uses kHalf; FromGguf uses kInterleaved for llama, mistral and mixtral and kHalf for the rest (decoder.cc L51–56, L66–79).

Two bases. When the config has rope_local_theta (Gemma 3), Create builds a second Rope with that base and no scaling, and sliding layers use it while global layers use the main one (decoder.cc L81–98, L307–311).

Fixture settings from make_llm_fixtures.py L47–98.

Mixture of experts

With experts, the FFN half is MoeFfn (decoder.cc L333–369). The router is one more WeightMatrix (ffn_gate_inp, [num_experts, hidden]); a MatMul scores every token against every expert, and then, token by token:

1. softmax the token's router scores over all experts;
2. pick the top k by repeated selection, ties going to the lower expert id, summing the chosen probabilities;
3. for each chosen expert, weight = p_e / total when normalize_expert_weights, else p_e; run that expert's MatMulGlu and down projection on the one token, and add weight × output into the residual stream.

Experts run one token at a time, so the gate buffer is one row wide (L174–177), and chosen ids sit in a stack array of 256, the most experts a model may have (L339–342). In a prefill that means one expert product per token per chosen expert rather than grouping tokens by expert; at decode, where n = 1, it is the same work.

OLMoE-1B-7B (64 experts, 8 active) is rml's MoE proof on a real checkpoint; Qwen3-30B-A3B does not fit the 16 GB budget (models.md L73–96).

The eight families

Sources: config.cc L292–367, decoder_test.cc L100–115, real_model_test.cc L130–134. The config.h header's family list (L13–20) names seven and leaves out OLMoE, which the parser supports.

How the goldens are made

tools/make_llm_fixtures.py writes one two-layer toy model per family, a few thousand parameters (vocab 64, hidden 32, 4 heads over 2 kv heads), twice: a Hugging Face directory with bf16 safetensors, and a GGUF file laid out as llama.cpp's converter lays it out (f16 matrices, f32 vectors, Llama q/k rows permuted, experts stacked, Gemma norms as 1 + w). The weights are exactly representable in both bf16 and f16, so both files hold the same model, and logits.json holds every prompt position's logits from a float64 numpy forward pass that follows the Hugging Face modeling code (L1–16, L31–45). The Llama variant also gets a Q8_0 GGUF.

real_model_test checks downloaded checkpoints against goldens from tools/make_real_model_goldens.py: tokenizer ids, the chat prompt, the top-8 logits at every prompt position within 2e-3, and an 8-token greedy continuation. It skips when RML_MODELS_DIR lacks the model (real_model_test.cc L1–6, L31–33). rml reports worst f32 logit errors of 1.4e-4 (SmolLM2-135M-Instruct), 1.1e-4 (TinyLlama-1.1B-Chat and Qwen3-0.6B), 1.7e-4 (Gemma 3 1B) and 1.3e-4 (OLMoE-1B-7B-Instruct), run in the development container; CI downloads only SmolLM2. Llama 3.1 8B at Q8_0 / Q4, the milestone model, has no golden yet and is planned (progress.md L58–59).

Tests

Two benchmarks go with them: attention_benchmark (one decode step of a Llama-3-8B layer's attention, per cache dtype) and decoder_benchmark (one decode step of a random-weight model with TinyLlama-1.1B's layer shape but two layers and an 8192-token vocabulary, Q8_0 and Q4_0, at 1 and 4 threads; L28–38). Their results are on Kernels. Comparisons against llama.cpp on the Mac are still planned (progress.md L71).