Tokenizers

In short: every tokenizer is a TokenizerSpec (vocabulary, token types, merges or scores, pre-splits, special ids) checked and indexed by Tokenizer::Create. Encoding cuts special tokens out of the text first, then spells each remaining segment: byte-level BPE (GPT-2, Llama 3, Qwen) pre-splits it into words and writes every byte as a printable character, so nothing is ever unknown; SentencePiece BPE (Llama 2, TinyLlama, Mistral, Gemma) writes spaces as "▁" and keeps the segment whole. A priority queue then applies the earliest-ranked merge (or highest-scoring piece), leftmost on ties, until none applies. The pre-split regexes are matched by hand rather than by a regex engine, against Unicode 15.1 category tables generated offline, and every split, encoding and decoding is checked against Python's regex, Hugging Face tokenizers and sentencepiece outputs committed as fixtures. No Jinja either: seven chat formats are recognized and written directly.

The concept is on /ml-models: tokenization. All of this lives in rml/llm as six libraries: tokenizer, pretokenizer, unicode (which compiles the generated unicode_tables), hf_tokenizer, gguf_tokenizer and chat_template (BUILD.bazel). The model they feed is on The decoder.

The spec and the two kinds

The header describes BPE in four steps (tokenizer.h L4–25): special and added tokens are cut out first and map straight to their ids; byte-level BPE pre-splits the rest and spells each byte as a printable character, while SentencePiece spells spaces as "▁" and prefixes one; merges run, lowest rank (highest score) first, leftmost on ties; and pieces map to ids, a SentencePiece piece outside the vocabulary falling back to one <0xXX> token per byte.

tokenizer.h L40–81. Create rejects an empty vocabulary, a types list of the wrong length, a score-merged SentencePiece spec without one score per token, out-of-range special ids, add_bos without a BOS id, and any merge whose two halves or result are not in the vocabulary (tokenizer.cc L103–151). It then builds the string-to-id map, the merge table keyed by the two ids, each token's raw bytes, and the list of control and user-defined tokens sorted longest first, so that the longest special wins when two share a prefix (L153–194).

The header says Create also checks that a byte-level vocabulary covers all 256 bytes (tokenizer.h L92–93); the code deliberately does not, because SmolLM2's vocabulary has no control bytes. Like Hugging Face, such a byte encodes to the unknown token if there is one and is dropped otherwise (tokenizer.cc L153–156).

The byte alphabet

Byte-level BPE needs a printable spelling for every byte. Bytes that print as themselves (! to ~, 0xA1–0xAC, 0xAE–0xFF) keep their code point; the other 68, in order, take U+0100, U+0101 and so on (tokenizer.cc L27–51). Space is the 33rd of them (bytes 0x00–0x20 come first), so it becomes U+0120 "Ġ", and newline (0x0A, the 11th) becomes U+010A "Ċ". The last lands on U+0143, which is why the inverse table has 0x144 entries. ByteToUnicode and UnicodeToByte expose the bijection (tokenizer.h L150–154), and a token's raw bytes are its spelling run back through it.

Pre-splitting

Byte-level BPE never merges across a pre-split boundary, so the split decides what a token can be: " hello" may be one token, "hello world" never is. Each model family defines its split as a regex; rml matches each known pattern by hand, alternative by alternative with the same leftmost-alternative semantics, instead of embedding a regex engine (pretokenizer.h L4–12).

The patterns are stored verbatim (pretokenizer.cc L14–19) so PreSplitFromPattern can recognize a tokenizer.json by exact string. The pieces of the hand matchers each mirror one alternative: Contraction (ASCII-case-insensitive for Llama 3), OptionalSpaceThenRun for ?X+, and TrailingSpace for \s+(?!\S)|\s+, a whitespace run minus its last character when a non-space follows and the run is longer than one (L48–93). Every code point is a letter, number, space or other and every pattern has an alternative for each, so a match always advances; a safety step of one code point covers the rest. Pieces are byte ranges of the input, so they cover it exactly, malformed UTF-8 included (L187–218). A list of splits applies each to every piece of the one before (L220–236).

The three patterns on the same text

From the committed reference splits, produced by Python's regex module on the original patterns (pretokenizer.json, written by make_pretokenizer_fixtures.py); pieces are shown between bars, · is a space, and each text is the start of a fixture string:

GPT-2 keeps digit runs whole and its contractions are case-sensitive; Llama 3 caps digit runs at three and matches 'M; Qwen2 isolates every digit. Llama 3's [^\r\n\p{L}\p{N}]?\p{L}+ lets one punctuation character lead a word, hence (x, and its [\r\n]* tail keeps newlines with the punctuation before them. The fixture has 25 such texts, also covering tabs, CRLF, combining marks, CJK, emoji and no-break spaces, and the test compares all 75 splits (pretokenizer_test.cc L33–50).

UTF-8 and Unicode classes

The patterns speak in code points and categories, the tokenizer in bytes; unicode.h bridges the two (unicode.h L4–10).

The ranges are generated: tools/make_unicode_tables.py walks all 0x110000 code points through Python's unicodedata, keeps the eleven categories the patterns use (mapped to eight enum values), and merges consecutive code points of one category into ranges, so the C++ library needs no Unicode dependency (make_unicode_tables.py L1–41, unicode_tables.h L9–33). The committed table has 2,351 ranges from Unicode 15.1.0 (unicode_tables.cc L1–7).

The merge loop

EncodeWord runs BPE over one spelled word (tokenizer.cc L325–406). The word starts as one symbol per unit (a byte's spelling, or a code point for SentencePiece), held as a linked list over byte ranges. Every adjacent pair with a merge goes into a min-priority queue keyed by priority, then by left position. Popping a candidate merges it only if it is still current: the left symbol must still point at the right one and both must have the lengths recorded when it was queued; otherwise it is stale and skipped. A merge grows the left symbol, empties the right, and queues the two new neighbouring pairs.

• Priority. With merge rules, a pair's priority is its rank, looked up by the two ids. Without them (SentencePiece from GGUF), the merged string itself is looked up and must be a normal or user-defined token; its priority is minus its score (FindMerge L299–323).
• Leftovers. A final symbol with no id becomes one byte token per byte under SentencePiece byte fallback (or unk_id for a byte without one), and unk_id or nothing otherwise (AppendPiece L408–421).

A worked example

tokenizer_test builds a byte-level vocabulary of the 256 byte symbols (ids 0–255, so id 97 is "a"), then "ab" (256), "abc" (257) and the control token <|end|> (258), with merges a b (rank 0) and ab c (rank 1) (tokenizer_test.cc L80–94). Encoding "abcab", which the GPT-2 split leaves as one word:

Decoding

Decode concatenates each token's raw bytes, skipping control tokens when asked. SentencePiece's own prefix space is removed from the first text token, since it is not part of the text (tokenizer.cc L423–443). Bytes, not characters, come out: one character can be split across tokens, so the header asks streaming output to buffer to a character boundary (tokenizer.h L23–25). TokenBytes, TokenText and TypeOf die on an out-of-range id.

Hugging Face tokenizer.json

ParseHfTokenizer reads a BPE model with byte-level pre-tokenization (GPT-2, Llama 3, Qwen2, SmolLM) or with SentencePiece-style "▁" spaces and byte fallback (Llama 2, TinyLlama, Mistral, Gemma); anything else is Unsupported (hf_tokenizer.h L4–11, hf_tokenizer.cc L237–293). It works component by component, flattening Sequences:

Exactly one of the two space styles must be present; both or neither is Unsupported. LoadHfTokenizer reads the file through a MappedFile (Model files) and prefixes errors with the path (L295–303).

GGUF tokenizer metadata

A GGUF file carries its tokenizer under tokenizer.ggml.* keys (gguf_tokenizer.h L4–17, gguf_tokenizer.cc L47–123):

Chat templates

A model's tokenizer config ships its prompt format as a Jinja template. Rather than run Jinja, rml recognizes the formats current open models use and writes them directly. The output leaves out the leading BOS, which the tokenizer adds, and the default system prompts and dates some templates inject (chat_template.h L4–20).

DetectChatFormat takes either a name ("chatml", "llama3", …) or a whole Jinja template and checks the markers in the order above, so the more specific ones win (chat_template.cc L163–189). FormatChat rejects roles other than system, user and assistant, and the [INST] and Gemma formats also require an optional first system message followed by strictly alternating user and assistant turns (L65–82, L191–213). With add_generation_prompt the prompt ends where the assistant's reply begins; the [INST] formats already end there.

Who uses it

examples/rml_chat is the one caller that puts the pieces together (chat.cc L56–144). A directory loads tokenizer.json and the Hugging Face chat template; a GGUF file loads the tokenizer from its metadata and the template from tokenizer.chat_template; a format given on the command line wins over either. The stop ids are the tokenizer's end-of-sequence ids, the model config's, and the format's turn terminator (<|im_end|>, <|eot_id|>, <|endoftext|>, </s> or <end_of_turn>) when it is in the vocabulary. Each reply formats the whole conversation, encodes it, and streams every non-control token's raw bytes to the caller as it is generated (L186–205). The generation loop itself is on Generation, and the program on Examples.

Tests and fixtures

tools/make_tokenizer_fixtures.py trains tiny tokenizers on a small mixed-language corpus with Hugging Face tokenizers (byte-level BPE with the Llama 3, Qwen2 and GPT-2 + digits splits, vocabulary 600, and SentencePiece-style BPE with byte fallback, 700) and with sentencepiece itself (score-based, 500, as GGUF stores it), and saves each with the ids the reference library gives a set of test strings (make_tokenizer_fixtures.py L1–13). The tests then need no Python.