Executing a graph

In short: Executor::Create runs shape inference, optionally groups runs of elementwise nodes into fused steps, compiles every other node into a kernel closure, and records for each step which values it is the last to read. Run checks the inputs against their recorded types, binds the constants, and calls the steps in order, dropping each value right after its last reader. Buffers freed that way go to the executor's BufferPool, so the next run reuses them instead of allocating. Shape ops return views, elementwise, softmax, norm and reduction ops run on vectorized threaded kernels, matmul and conv2d go through the GEMM, and a constant matmul weight is packed once at load.

The graph it runs is described on The rml-graph IR; what fusion and the fast kernels do inside a step is on Lowering and fast CPU. The executor is a general model runner, not the allocation-free LLM decode loop, which stays in rml/llm (graph_ir.md L118–119).

The interface

Run is const and keeps every per-call value in locals, so the compiled steps are never modified by a run.

Building the step list

Executor::Create does all the work that does not depend on input data, once:

1. Check. InferTypes confirms every op, attribute and recorded type (L57), so an executor never exists for a graph that would fail later.
2. Pin. Graph outputs and constants are marked as never released (L60–68).
3. Fuse. With fuse on, FindFusionGroups picks runs of two or more consecutive elementwise nodes with one output shape, connected or not (L70–73). Each group becomes one step labelled fused(...) with a kernel from MakeFusedKernel, whose outputs are every member value read outside the group, and the nodes it covers are skipped (L105–114).
4. Fold conv biases. Also with fuse on, a bias-free conv2d whose only reader is the next node, an add of an f32 per-channel constant shaped [O], [O, 1, 1] or [1, O, 1, 1], becomes one step labelled conv2d+add: the constant is passed as the conv's bias input, saving a pass over its output (FoldableConvBias, L86–104). JAX and TensorFlow export a conv bias that way; a conv whose output is a graph output or is read elsewhere stays two steps.
5. Compile. Every other node becomes one step whose kernel comes from CompileOp, which is also told which inputs are graph constants so it can prepare them once (L115–127).
6. Plan releases. While building, each step overwrites last_reader for every value it reads. Afterwards each value that is read and not pinned is added to the release list of its last reader (L129–133).

A run

RunImpl, behind both Run and RunProfiled:

1. Installs the executor's pool with a ScopedBufferPool for the length of the run (L156). From here every Tensor::Empty on this thread draws from that pool (BufferPool).
2. Checks the input count and each input's dtype and shape, naming the value on a mismatch, and binds the inputs and constants into a vector indexed by value id (L157–171).
3. For each step: gathers its inputs, calls the kernel, stores its outputs, then resets every value in its release list to an empty tensor (L175–185). A kernel error returns at once. With profiling on, the step's wall time is added to its slot (L186–190).
4. Returns each graph output made contiguous (L194–197), so a view never leaks out of the executor.

Value lifetimes

Values are reference-counted tensors, so resetting a slot frees the buffer only when nothing else holds it. A view keeps its base alive, and an output that is a view of an intermediate keeps that intermediate's storage until the output is copied at the end. Peak memory therefore follows the graph's live set, not its total size (graph_ir.md L115–117). When the last reference to a pooled buffer goes, the buffer returns to the executor's pool, sized by its byte count, and the next allocation of that size, in this run or the next, takes it back.

How ops map onto kernels

Each op's compiler in ops.cc returns the kernel as a closure over its attributes, so a run does no attribute parsing or dispatch beyond one call per step. The fast paths work on contiguous f32 data: a strided input is copied to contiguous first, which costs less than the generic strided loops (ops.cc L125–126). The kernels named *Fast are in rml/graph/kernels.h.

Views

Shape ops cost no copy (graph_ir.md L117–118). Each returns a tensor sharing its input's storage with new shape and strides, as described on Tensors and dtypes:

A view is paid for by its reader: the next fast kernel copies a strided input to contiguous, and the executor makes graph outputs contiguous before returning them. A chain of shape ops therefore costs at most one copy, made by whichever op first needs contiguous data.

rml-graph-run

examples/graph_run builds //examples/graph_run:rml-graph-run (BUILD.bazel L27), which runs one exported model and prints one line of JSON. Its logic lives in a library, graph_run.h, so the test can drive it without the command line (graph_run.h L4–64).

main times LoadGraph plus Executor::Create as load_ms, runs once, then optionally checks, benchmarks and profiles (main.cc L109–161). Time runs the warmup, then times each call with steady_clock (graph_run.cc L87–100), and Summarize reports iterations, mean, min, p10, median and p90 by nearest rank. ProfileOps sums RunProfiled's step times by step label, so all matmul steps share one row and each distinct fused program has its own, sorted slowest first with the node count each row covers (L102–121). The JSON keys are model, threads, load_ms, then max_abs_error, the latency fields and profile when present (L123–144). tools/bench_frameworks.py reads this line for its rml cells (Benchmarks against the frameworks).

bazel run --config=opt //examples/graph_run:rml-graph-run -- \
  $PWD/rml/graph/testdata/torch_cnn --inputs $PWD/rml/graph/testdata/torch_cnn.golden.safetensors \
  --check --bench 100 --threads 1

That invocation is from rml's README.

The executor benchmark

executor_benchmark.cc registers one Google Benchmark per .json in $RML_GRAPH_DIR, or in rml/graph/testdata when it is unset, for example the models/graphs directory that tools/bench_frameworks.py --export-only fills (L1–3). Each graph is loaded and compiled once at registration, and graphs that fail to load are skipped. The timed loop calls Run on inputs filled with 0.5, so it measures steady-state runs with the buffer pool warm, reported in microseconds (L22–56). rml commits no results for it; published timings come from rml-graph-run through the framework benchmark.

Tests