Models and memory

In short: rml's budget is a 16 GB Apple Silicon Mac with the inference process under about 11 GB peak RSS, and docs/models.md works the numbers for each target from its config.json: weights at a given bits per weight, plus a KV cache, prefill scratch and runtime overhead. Llama 3.1 8B fits at Q8_0 (about 10.2 GB, tight) and Q4_K_M (about 6.6 GB, the performance target) but not bf16; Qwen3 8B is the second dense architecture; Qwen3-30B-A3B does not fit, so OLMoE-1B-7B proves the MoE path. In the code, ParameterCount reproduces the parameter counts and its tests pin them. Five real checkpoints, from SmolLM2-135M to OLMoE-1B-7B, pass golden tests; the 8B milestone models and every measurement of peak RSS are still planned.

The models page sits beside the rest of rml's planning documents: the milestone in progress.md L5–8 is that rml-chat loads Llama 3.1 8B Instruct at 4-bit by mmap, stays under about 11 GB peak RSS, matches the reference top-1 tokens and decodes within 1.5× of llama.cpp, and the README's goal says the same (README.md L8–10). How weights stay mapped is on Model files; the block formats behind "bits per weight" are on Quantization; the concepts are quantization and the KV cache.

The budget and the formula

The process gets about 11 GB so macOS, the window server and a browser keep about 5 GB. Weights are mmapped and only count toward RSS as pages are touched, but decode touches every weight on every token, so in steady state the whole file is resident: the document plans as if it all counts (models.md L3–6). Every table on the page is built from the same parts (L8–22):

Figures are decimal GB, and the document asks that they be checked against the downloaded file size before anyone relies on them (L21–22). Two of its assumptions run ahead of the code. The prefill row assumes 512-token chunks, but Session::Generate runs the whole unmatched prompt in one Forward at the pin, so chunked prefill is planned (see Generation). And no code at the pin calls madvise, so dropping each layer's pages after a full-precision golden run (L41–43) is planned too.

The parameter count in code

ParameterCount(const ModelConfig&) is the same sum, counting a tied head once (config.cc L495–506): per layer, the Q, K, V and output projections, any q/k/v bias and QK-norm weights, three hidden × intermediate matrices per expert (one expert for a dense model) plus the router, and two norm vectors (four with Gemma 3's sandwich norms); then the final norm and the embedding, doubled when the head is untied. The config tests pin it for each family, so the numbers docs/models.md quotes are checked in CI:

Qwen3 8B and Qwen3-30B-A3B have no config in the tests; their counts (8.19 B and 30.5 B) come only from the document.

Milestone: Llama 3.1 8B Instruct

32 layers, hidden 4096, 32 query heads and 8 KV heads (GQA 4:1), head_dim 128, SwiGLU FFN 14336, vocab 128256, an untied head, RoPE θ = 500000 with llama3 frequency scaling (factor 8, low/high factors 1 and 4, original context 8192). P = 8.03 B, of which the embedding and the untied head are 0.53 B each (models.md L24–31); the same scaling fields are what ParsesLlama31RopeScaling checks.

The consequences (L41–45): bf16 cannot be the runtime format, so full-precision checks on the 8B model either page through the mmap layer by layer or run Q8_0 against bf16 reference logits; Q8_0 fits with about 0.8 GB to spare and an int8 KV cache adds headroom; Q4_K_M is the main performance target. rml can run both quantized formats: DotActivationType pairs Q8_0 and Q4_0 weights with Q8_0 activations, and Q4_K, Q5_K and Q6_K (the mix inside a Q4_K_M file) with Q8_K (dot.cc L285–297).

Worked example: the KV cache

With the Llama 3.1 8B shape, one position in one layer stores kv_heads × head_dim = 8 × 128 = 1024 keys and as many values. At f16 and an 8192-token context:

2 (K, V) × 32 layers × 8 kv_heads × 128 head_dim × 8192 tokens × 2 bytes
  = 1,073,741,824 bytes ≈ 1.07 GB

rml's KvCache holds exactly that per position and layer as a row of row_bytes() (kv_cache.h L55–56): 4096 bytes at f32, 2048 at f16. Its int8 mode is not quite half of f16, because each group of quant_group() channels (at most 32) carries one f32 scale, 1.125 bytes per entry (L13–18): a 1152-byte row here, so about 0.60 GB for 8K tokens where models.md L44 rounds the int8 cache to 0.54 GB by leaving the scales out. The cache grows a 16-token page at a time out of an arena reserved up front, and pages it never touches are never made resident. KvCache::Create defaults to f32 (L40–43), but rml-chat asks for f16 and a 4096-token context unless told otherwise (main.cc L48–49), which is 0.54 GB for this model. The layout is drawn on The decoder.

Why the memory number is also the speed number

Decode is memory bound: each token reads every weight once. At Q4_K_M (about 4.9 GB a token) the ceiling is about 14 tok/s on an M1 (68 GB/s), 20 on an M2 or M3 (100 GB/s) and 24 on an M4 (120 GB/s), and achieved GB/s as a fraction of that is meant to be the headline benchmark (L47–49). That benchmark, beside llama.cpp on the same file, is planned (progress.md L71); see Kernels.

Second architecture: Qwen3 8B

36 layers, hidden 4096, 32 query and 8 KV heads, head_dim 128, FFN 12288, vocab 151936, an untied head, per-head RMSNorm on Q and K before RoPE, θ = 1000000; P = 8.19 B (models.md L51–56).

The table is L58–63. Qwen3 differs from Llama only in config-driven ways, which is the point of choosing it: the config-driven DecoderLM has to absorb QK-norm, a different θ and no RoPE scaling with no new model class (L65–66); how it does is on The decoder. The document also flags that newer releases (Qwen3.5, newer Gemma and Llama generations) could not be checked from a sandbox without Hugging Face access, and asks that the table be redone before Phase 6, preferring a newer dense ~8B model if one fits (L68–71).

Mixture of experts

A sparse model touches only its active experts per token, but the whole file still has to be stored somewhere. Mixture of experts explains the routing; here is what it does to the budget (models.md L73–96):

Streaming experts from SSD through the page cache is rejected: which experts a token touches changes every token, so the resident set still grows toward the whole file and the OS would thrash (L83–85). OLMoE won out without special handling; its real-weight golden passes, and a Qwen1.5-MoE golden is listed only as a later step (progress.md L293).

Status per model

Three sources describe where each model stands, and they are not equally current. The README's table (README.md L249–256) lists only the targets and test models; progress.md L58–59 lists the real-weight goldens that have run; and the code says what is actually checked: real_model_test's case list (real_model_test.cc L130–134) and the golden files under rml/llm/testdata/goldens.

The worst-error figures are the ones progress.md L58–59 reports, all in f32 and all run in the development container; the test itself allows 2e-3 per logit (real_model_test.cc L33). The README's status line says phases 1–7 are done apart from the macOS-only work: real-weight golden tests, NEON and Metal kernels, and the llama.cpp comparison (README.md L15–16).

Test models

SmolLM2-135M uses the Llama block with a tied head and GQA 3:1, so it runs the same code path as the 8B target at a size CI can download and run in seconds (models.md L98–107). The toy configs are generated by tools/make_llm_fixtures.py and live in the repository; how they and the real goldens are made is on Build and testing.

Getting the weights

Weights live in a gitignored models/ directory and are never committed (AGENTS.md L40). tools/download_model.py fetches a Hugging Face checkpoint into models/<name> using only the standard library:

Only safetensors are downloaded; GGUF files such as the README's llama-3.1-8b-q4_k_m.gguf are fetched by hand. CI uses the script for one model, SmolLM2-135M-Instruct at a pinned revision, cached between runs (ci.yml L48, L56–63).

Tests