rml

rml architecture

A source-linked map of rml, a machine learning library written from scratch in C++20 with Bazel that is becoming a layered inference stack: models defined in PyTorch, JAX and TensorFlow, exported into rml's own graph IR and run on its CPU kernels.

On this page
  1. Where rml is going
  2. The library's goals
  3. Module layering
  4. Module map
  5. Pages
  6. Inference stack
  7. Foundations
  8. Learning
  9. Classical & retrieval
  10. LLM runtime
  11. Project

Where rml is going

rml is becoming a layered, MLIR-style inference stack. Every neural model on the ML models reference is to be defined in PyTorch, JAX and TensorFlow, exported into rml's own graph IR, and checked against each framework's outputs. Inside rml, that graph lowers to a fused, buffer-planned, threaded kernel-level plan with SIMD GEMM, aiming to beat the frameworks' eager and compiled modes on CPU, with every model benchmarked against them (the plan in docs/progress.md).

At this commit the stack runs end to end: a 19-model zoo in all three frameworks, the direct and StableHLO export routes, the rml-graph format with its loader and executor, fused elementwise programs below it, vectorized threaded kernels and a packed SIMD GEMM, and a benchmark of every model against every framework, which rml now leads on average at every batch and thread count. Keeping small ops on the calling thread is the one item still open on rml's checklist. The inference stack pages follow it level by level.

The library's goals

rml's README gives it two goals. The first is to teach by implementing: every concept on the ML models reference gets a clean, readable, tested implementation. The second is to run modern LLMs fast on a laptop: load open-weight decoder-only models such as Llama 3.1 8B and Qwen3 8B by mmap, and decode within about 1.5× of llama.cpp on a 16 GB Apple Silicon Mac. There are no ML dependencies.

The two goals meet in the tensor type (docs/architecture.md). Training code is written for clarity; the decode loop calls fused kernels that are tested against the readable versions. These pages describe one commit, version 0.1.0, and mark anything rml's docs describe but its code doesn't contain yet as planned.

Module layering

Each module is a directory under rml/, and may depend only on modules below it (the layering table). Every box links to the page that explains it, or to the module's pinned source until that page exists. Dashed boxes are planned and have no code at this commit.

rml module layering Seven layers, lowest first. core holds tensors, dtypes, ops, errors, memory and threads. Above it, linalg and random. Above them, autograd. Then nn, losses and optim, beside data, metrics and classical. Then retrieval. Then quant, io, and two planned modules, backend and profile. At the top, side by side, llm (tokenizers, the decoder, the KV cache and sampling) and graph (the loader and executor for models exported from PyTorch, JAX and TensorFlow). A module depends only on modules below it. llm tokenizers · DecoderLM · KV cache · sampling graph rml-graph loader · fusion · kernels · executor quant GGML blocks, dequantize, dot backend planned: kernel dispatch io mmap, safetensors, GGUF profile planned: timers, RSS retrieval flat and IVF nearest-neighbour search nn modules, layers losses loss functions optim optimizers data datasets, splits metrics scores classical 11 models autograd Variable, backward, grad check linalg GEMM, LU, QR, Cholesky, eigen, SVD random Philox, distributions core Tensor · DType · Shape · ops · Status · allocator · ThreadPool

Module map

Every directory under rml/ at this commit, its headers (tests and benchmarks left out), and the page that covers it. examples/ holds 32 programs that parse flags and print; the logic they run lives in these libraries.

ModuleFilesPage
rml/core allocator, buffer_pool, check, dtype, ops, shape, status, strided, tensor, thread_pool Tensors and dtypes, Errors, memory and threads, Kernels
rml/linalg cholesky, eigen, gemm, lu, matmul, matrix, norms, qr, svd Linear algebra, Kernels
rml/random distributions, philox Random numbers
rml/autograd function, grad_check, ops, variable Autograd
rml/nn activation, attention, conv, dropout, embedding, init, linear, module, normalization, recurrent, transformer Neural network layers
rml/losses classification, contrastive, reduction, regression, variational Losses and optimizers
rml/optim accumulation, adam, clip, optimizer, rmsprop, schedule, sgd Losses and optimizers
rml/data dataset, preprocessing, split Data and metrics
rml/metrics calibration, classification, inputs, ranking, regression Data and metrics
rml/classical common, decision_tree, gradient_boosting, kmeans, knn, linear_regression, logistic_regression, naive_bayes, pca, random_forest, svm Classical models
rml/retrieval ivf, similarity Retrieval
rml/quant blocks, dequantize, dot, ggml_type Quantization, Kernels
rml/io checkpoint, gguf, mapped_file, safetensors, safetensors_model Model files
rml/llm attention, chat_template, config, decoder, generation, gguf_tokenizer, hf_tokenizer, kernels, kv_cache, pretokenizer, rope, sampler, speculative, tokenizer, unicode, unicode_tables, weight_source Tokenizers, The decoder, Generation, Kernels
rml/graph executor, fusion, graph, kernels, ops, shape_inference The inference stack, Exporters and the model zoo, The rml-graph IR, Executing a graph, Lowering and fast CPU, Kernels
rml/copts rml Build and testing

Pages

Inference stack

  • The inference stack

    Where rml is headed: framework models exported into rml-graph, lowered to a fused kernel plan and benchmarked against PyTorch, JAX and TensorFlow, level by level.

  • Exporters and the model zoo

    How tools/rml_export traces PyTorch, JAX and TensorFlow models into rml-graph, the passes it runs, the shared zoo, and the golden fixtures.

  • The rml-graph IR

    The exported model format: JSON values, constants and nodes over mmapped safetensors, the op set, loader validation and shape inference.

  • Executing a graph

    The executor's order and value lifetimes, how each op maps onto rml kernels, the rml-graph-run tool, and the executor benchmark.

  • Lowering and fast CPU

    The kernel-level layer below rml-graph: fused elementwise programs, threaded vectorized kernels, buffer reuse and SIMD GEMM, each tied to the gap it targets.

  • Benchmarks against the frameworks

    How each model is timed in PyTorch, JAX and TensorFlow, eager and compiled, and in rml, and what the published results show.

Foundations

  • Tensors and dtypes

    DTypes and their storage, shapes and strides, broadcasting, the strided row walker, Tensor views versus copies, the f32 reference ops, and RML_CHECK.

  • Errors, memory and threads

    Status and Expected instead of exceptions, aligned allocation with its counters, and the ThreadPool whose ParallelFor gives reproducible results.

  • Linear algebra

    Matrix, matmul and the blocked GEMM kernel, and the LU, QR, Cholesky, eigen and SVD decompositions plus norms.

  • Random numbers

    The Philox counter-based generator and the distributions sampled from it.

Learning

  • Autograd

    Variables, functions and the backward pass over a dynamic graph, the differentiable ops, and numerical gradient checking.

  • Neural network layers

    Module and every layer: linear, activations, initialization, convolution, embedding, normalization, dropout, recurrent, attention and transformer blocks.

  • Losses and optimizers

    Classification, regression, contrastive and variational losses, and SGD, Adam, RMSprop, schedules, clipping and gradient accumulation.

  • Data and metrics

    Datasets, preprocessing and splits, and the classification, regression, ranking and calibration metrics.

Classical & retrieval

  • Classical models

    The fifteen classical estimators, from linear regression to gradient boosting, and the tests that hold them to scikit-learn fixtures.

  • Retrieval

    Similarity functions and the inverted-file (IVF) vector index.

LLM runtime

  • Quantization

    GGML block types, dequantization, and the quantized dot products the decoder runs on.

  • Model files

    Memory-mapped files, safetensors, GGUF and checkpoints, and how weights stay mapped rather than copied.

  • Tokenizers

    The tokenizer interface, pretokenization and Unicode tables, Hugging Face and GGUF tokenizers, and chat templates.

  • The decoder

    Config, weights, one layer's forward pass, attention and the KV cache, RoPE, mixture of experts, and the eight model families.

  • Generation

    Samplers, the generation loop, speculative decoding, the rml_chat example and the decode loop end to end.

Project

  • Kernels

    What docs/kernels.md describes: the GEMM, quantized dot, attention and fused decoder kernels that exist, and the ones still planned.

  • Models and memory

    Target models and the memory math behind them, and the status of each model family.

  • Examples

    Every example program, what it builds from the library, and the concept it demonstrates.

  • Build and testing

    Bazel modules and copts, GoogleTest and Google Benchmark, golden LLM tests, the fixture tools, and CI.