Build and testing

In short: MODULE.bazel pulls in rules_cc, platforms and nlohmann/json, plus GoogleTest and Google Benchmark as dev dependencies; every target goes through the rml_cc_* macros in rml/copts/rml.bzl, which add the warning flags with -Werror and link gtest_main or benchmark_main. At the pin there are 125 test files with 837 TEST, TEST_F and TEST_P definitions, 96 death tests and seven benchmark programs. The LLM decoder is checked against tools/llm_reference.py, a numpy forward pass that follows Hugging Face's modeling code: on committed two-layer toy models for all eight families in safetensors and GGUF, and on five downloaded real checkpoints when they are present. CI builds and tests everything on macOS (Clang: opt, ASan, UBSan) and Ubuntu (GCC and Clang opt, GCC ASan), runs the SmolLM2-135M golden on both, and checks formatting. Python is never needed to build or test; it only regenerates fixtures.

The rules this page traces are in rml's AGENTS.md: Bazel with bzlmod and no CMake; no ML dependencies; Python only in tools/; a scalar reference and a comparison test for every optimized kernel; GoogleTest and Google Benchmark as the only test frameworks; fixed seeds; and every differentiable op with a numerical gradient check (AGENTS.md L50–67, L127–153). How the kernel references are tested is on Kernels; the gradient checks are on Autograd.

The Bazel module

The README's dependency table lists each with its license, all permissive (README.md L301–313); NumPy, PyTorch, JAX, TensorFlow and torch-mlir appear there too but are used only by scripts in tools/ and are never built or linked.

Configs

The rml_cc_* macros

No BUILD.bazel in rml calls cc_library directly. The four macros in rml.bzl add one flag list to first-party code only, so third-party headers are not held to it:

RML_COPTS = ["-Wall", "-Wextra", "-Wpedantic", "-Wshadow",
             "-Wconversion", "-Wno-sign-conversion", "-Werror"]

The flags are the reason so much of rml's code carries explicit static_casts: -Wconversion rejects implicit narrowing (an int64_t into an int, a double into a float), while -Wno-sign-conversion leaves signed–unsigned mixing to review. Each package also turns on the layering_check and parse_headers features, so a file may only include headers of targets it lists in deps, and every header must compile on its own (AGENTS.md L121–125; for example examples/mlp/BUILD.bazel L3–9). Formatting is Google style at 100 columns (.clang-format), checked in CI.

GoogleTest and Google Benchmark

Tests sit next to the code they test as foo_test.cc and live in namespace rml { namespace { … } }. The house style prefers EXPECT_* so one run reports every failure, keeps ASSERT_* for when the rest cannot run, compares floating point with EXPECT_FLOAT_EQ or EXPECT_NEAR and a stated tolerance, sweeps shapes and dtypes with TEST_P, and seeds every randomized test (AGENTS.md L127–140). At the pin:

Benchmarks are Google Benchmark programs run with bazel run --config=opt //rml/<module>:<name>; they are testonly binaries, so bazel test //... builds but never runs them. Their numbers are recorded in docs/kernels.md and quoted on Kernels, Linear algebra and Lowering and fast CPU.

Death tests

Errors cross API boundaries as Status or Expected<T>; a programmer error, such as a shape mismatch or a bad index, is an RML_CHECK that prints and aborts instead of throwing, so it is safe on hot paths (check.h L4–6, check.cc L9–13). RML_DCHECK compiles to nothing under NDEBUG (check.h L23–29). Death tests hold each check to its message:

EXPECT_DEATH(t.data<std::int32_t>(), "type does not match");
EXPECT_DEATH(Matrix2x3().Reshape(Shape{4, 2}), "number of elements");
EXPECT_DEATH(tt.View(Shape{6}), "contiguous");

(tensor_test.cc L52–89.) Suites made only of death tests are named *DeathTest, as GoogleTest recommends so they run first, for example DropoutDeathTest and ScheduleDeathTest. How Status and the checks divide the work is on Errors, memory and threads.

Golden LLM tests

The decoder cannot be checked against a hand-computed answer, so it is checked against an independent implementation: tools/llm_reference.py, a numpy forward pass that follows the Hugging Face transformers modeling code for every family rml loads, takes weights named as in a Hugging Face checkpoint, and runs in float64 by default (L1–7, L108). Two tools feed it, and two tests read what they write.

Toy models: every family, every format

make_llm_fixtures.py builds a two-layer model a few thousand parameters in size per variant (vocab 64, hidden 32, four heads, two KV heads) and saves it twice: as a Hugging Face directory with bf16 safetensors and as a GGUF laid out the way llama.cpp's converter lays it out (f16 matrices, f32 vectors, q/k rows permuted for Llama-family models, experts stacked, Gemma's norms stored as 1 + w). The weights are exactly representable in both bf16 and f16, so both files hold the same model; the llama variant also gets a Q8_0 GGUF whose reference uses the dequantized weights (L2–16, L30–44). The eight variants are llama, qwen2, qwen3, mistral, mixtral, qwen3_moe, gemma3 and olmoe (L47–101).

DecoderGoldenTest runs 17 cases, each family from both files plus the Q8_0 GGUF, twice: once as a whole-prompt prefill comparing every position's logits, and once as a three-token prefill followed by one token at a time through a cache with four-token pages, so the sequence spans several pages. The bound is tol + tol · |expected| with tol = 2e-4, loosened to 3e-2 for Q8_0 because rml dots quantized weights against int8-rounded activations, as llama.cpp does, which the float reference does not model (decoder_test.cc L38–98).

Real checkpoints

make_real_model_goldens.py takes a directory name under models/ and a chat format, and writes rml/llm/testdata/goldens/<name>.json (L130–168):

The reference runs in float32 here, reading bf16 and f16 weights lazily, so checkpoints too large to hold in float32 still fit (L102–117). The script knows five chat formats, chatml, zephyr, tulu, llama3 and gemma (L53–83); the committed goldens use four of them:

llama3 waits for the Llama 3.1 8B golden, which is planned for the Mac. The worst errors each model reached and the status of each are on Models and memory; the test cases are listed at real_model_test.cc L130–134.

The fixture tools

Everything under testdata/ is generated by a script in tools/ and committed when small, so the C++ tests never run Python (AGENTS.md L152–153). Each script checks rml against an independent implementation:

CI

One workflow, .github/workflows/ci.yml, runs on every push to main and every pull request, cancelling a superseded run on the same ref (L3–10). It has three jobs:

What CI does not cover: the tsan config is defined but has no job; the four larger real-model goldens run only where someone has downloaded the checkpoints; and benchmarks are never run in CI, so every number on these pages comes from a developer's run recorded in rml's docs. rml's guidance says to run --config=asan and --config=ubsan locally before pushing anything that touches memory (AGENTS.md L140), and that a change that could not be run in the Linux development container, such as NEON, Accelerate or Metal code or a large-model test, is verified on CI's macos-14 runners or on the Mac and noted in docs/progress.md (AGENTS.md L101–103).