Tensors and dtypes

In short: a Tensor has a shared Storage, an element offset, a Shape, Strides and a DType. Views such as transposes, slices and expands only change the offset, shape and strides. One loop, ForEachRow, walks strided data row by row for conversions and the ops; plain copies merge axes first and copy two-axis blocks, threaded when large. Broadcasting gives a stretched axis stride 0. The reference ops are f32-only, and quantized dtypes are stored but never computed on here. Contract violations abort through RML_CHECK.

This page covers rml/core apart from the allocator, status and thread pool, which are on Runtime. For the ideas behind it, see Tensors and shapes and Quantization in the ML models reference.

The seven dtypes

DType is a uint8_t enum. A table indexed by it holds each type's name, how many elements a storage block has, how many bytes that block takes, and two flags. Plain types have one element per block. The two GGML formats pack 32 elements into a block. Storage sizes are always computed from this table, so a tensor's byte size is numel / block_elems × block_bytes.

f16 and bf16 rounding

Both half types are a uint16_t of bits with FromFloat and ToFloat. They differ in which bits of an f32 they keep. bf16 keeps f32's 8-bit exponent and cuts the mantissa to 7 bits, so it has f32's range at lower precision. f16 has a 5-bit exponent and a 10-bit mantissa, so it is more precise but overflows above 65504.

Because there are only 65,536 halves, the tests check every one. EveryHalfRoundTrips converts each half to f32 and back and expects the same bits. Other tests compare against the compiler's own _Float16 conversion and check the bulk path against ToFloat for every half. The Mixed precision page explains why bf16 replaced f16 for training.

Quantized blocks

rml stores two GGML block formats in core so that a tensor can hold quantized weights straight out of a GGUF file. A block is 32 consecutive elements of a row with one f16 scale d. The formulas are in the header comments.

core only stores and moves these blocks. Quantizing, dequantizing, the K-quant formats and the dot products are on Quantization. Since a block can't be split, every view of a quantized tensor must keep whole blocks along the last axis (see Views and copies).

Shape and strides

Broadcasting

rml follows NumPy. Shapes are aligned on the right, and each pair of dimensions must be equal or one of them must be 1. A missing leading dimension counts as 1.

Stride 0 is how broadcasting avoids copies. Tensor::Expand uses it to make a view, and the binary ops use it to read both operands at the output shape. Writing through such a view would write the same element several times, so the header says expanded tensors must not be written.

The row loop

internal::ForEachRow<N> is the one loop for strided data in core. Dtype conversions, quantized copies, the binary ops and the reductions all go through it; plain copies use their own blocked path (see below). It takes a shape and N stride lists, one per operand. It then calls fn(offsets, n, inner_strides) once for each row of the innermost axis, where offsets[k] is where the row starts in operand k.

• The outer axes advance like an odometer: add the axis stride, and when the index reaches the axis size, subtract stride × size and carry to the next axis out (L37–46). There is no division or modulo per element.
• A rank-0 tensor is a single row of one element, and a shape with zero elements makes no calls (L23–28).
• The callback's inner loop is a plain counted loop over n elements with a fixed stride. When every inner stride is 1, the compiler can vectorize it.

Storage and Tensor

Views and copies

Every view method copies the Tensor (a shared_ptr copy) and edits the offset, shape or strides. SharesStorageWith tells you whether two tensors are views of one buffer. Methods that need whole blocks check for them on quantized tensors.

StridedCopy backs Contiguous, Clone and CopyFrom, and has three paths. When both sides are contiguous it is one memcpy. Quantized data is copied a row at a time with memcpy, and rows must be contiguous because a block has no element stride. Everything else first goes through CoalesceAxes, which drops size-1 axes and merges adjacent axes that both sides step through as one, so most views shrink to two or three axes. CopyBlock then copies the innermost two axes as one block: each row is a memcpy when both inner strides are 1, a fill_n when the destination stride is 1 and the source stride is 0 (a broadcast), and a strided loop otherwise. CopyPlain splits the outer blocks over the default thread pool once a copy has 2¹⁶ elements or more and at least two blocks.

The old loop handled one innermost row per callback on one thread. Upstream measured the U-Net graphs at batch 64 on 4 threads: the nearest-2× upsample's reshape went from 8.2 to 1.7 ms for PyTorch and 7.5 to 1.6 ms for TensorFlow, and concat from 4.7 to 2.7 and 9.0 to 4.2 ms (kernels.md L230–256).

Reference ops

ops.h is the f32 reference library. These ops are written to be obviously correct, not fast. Autograd builds on them, and the fast kernels on Kernels are tested against them. Each op returns a new contiguous tensor and aborts on a non-f32 input.

Three implementation details carry most of the file:

• Binary computes the broadcast shape, builds three stride lists (output, then a and b through BroadcastStrides), and runs the element function in ForEachRow<3>. All six binary ops and the comparisons share it.
• ForEachReducedRow permutes the reduced axis to the end as a view, then walks rows with ForEachRow<1>. The remaining axes keep their order, so the rows come out in the output's row-major order and need no index math.
• RowSum accumulates in double, so summing a long row doesn't lose the low-order bits a float accumulator would.

Softmax is three lines built from the other ops: Sub(x, Max(x, axis, true)), then Exp, then Div by the kept-dimension Sum. Subtracting the max keeps exp from overflowing and doesn't change the result. The keep-dim reductions broadcast back against the input through Binary.

RML_CHECK

rml has no exceptions. A broken contract, such as a shape mismatch, a wrong dtype or an out-of-range axis, is a programming error. RML_CHECK(cond, message) calls CheckFailed, which prints the file, line, condition and message to stderr and calls abort(). The failing branch is marked [[unlikely]]. RML_DCHECK compiles to nothing under NDEBUG. It guards the per-element bounds checks in At, where a check in release builds would cost too much.

Failures a caller can recover from, like running out of memory or a malformed file, go through Status and Expected<T> instead. Those are on Runtime.

Tests