ml-models

Machine learning, from the math up

A reference for learning, and relearning, any part of machine learning: the math under it, how to split and score data, the classical models, neural networks, the Transformer and the language models built from it, generative models, retrieval, and the systems that train and serve them. Every code link points at the exact lines of one pinned release of PyTorch, scikit-learn or XGBoost.

On this page
  1. The learning path
  2. The map
  3. Shared vocabulary
  4. Pinned releases
  5. Pages by area
  6. Mathematical foundations
  7. Core ML concepts
  8. Data and evaluation
  9. Classical ML
  10. Neural network fundamentals
  11. Model architectures
  12. Generative modeling
  13. Language models
  14. Retrieval and recommendation
  15. Training systems
  16. Inference and deployment
  17. Specialized learning
  18. Upcoming models
  19. Classic machine learning
  20. Deep learning foundations
  21. The transformer era
  22. Current state of the art

Every model answers the same question: given examples of inputs and the outputs we want, which function maps one to the other, and how do we find it? A model fixes the shape of that function, a loss scores how wrong it is, an optimizer adjusts it, and an evaluation on data it never saw says whether it learned anything. The pages below take that one idea from a straight line through a cloud of points to a model with billions of parameters.

Each page follows the same order: the problem the idea solves, the intuition in plain language, the mechanics with shapes and formulas, a worked example with real numbers, the implementation linked line by line to library source, its tradeoffs, and its connections to what comes before and after. Each page is marked intro, core or advanced by where it starts, but every page goes all the way down. Start with the learning path if this is all new, or jump to any area.

The learning path

Seven steps for someone starting from zero. Pages not written yet show as plain text.

  1. Tensor shapes and matrix multiplication
  2. Backpropagation and optimization
  3. Embeddings and attention
  4. The full Transformer block
  5. Training versus inference
  6. Evaluation and data leakage
  7. Classical baselines

The map

Three building blocks (top) make up the encoder-decoder Transformer of 2017 (middle). Every model family below it keeps those blocks and changes what goes in, what comes out, or which part is kept. All of them are trained by the same loop (bottom). Each box links to its page, or to the pinned PyTorch source while the page is being written.

Map of the model architectures this section covers Three building blocks, attention, positional encoding, and LayerNorm with residual connections, combine into the encoder-decoder Transformer. From the Transformer come five families: decoder-only language models like GPT keep the decoder with a causal mask; encoder-only models like BERT keep the encoder; the Vision Transformer feeds image patches to the encoder; mixture-of-experts models replace the feed-forward block with routed experts; and diffusion U-Nets put attention layers inside a convolutional network. All of them are trained by the same loop: cross-entropy or mean-squared-error loss, AdamW, a learning-rate schedule and mixed precision. BUILDING BLOCKS Attention softmax(QKᵀ/√d)·V, heads, masks Positional encoding sinusoidal, learned, RoPE LayerNorm and residuals x + f(norm(x)) THE ORIGINAL The Transformer (2017) encoder stack + decoder stack, nn.Transformer FAMILIES BUILT FROM IT Decoder-only GPT: causal LM Encoder-only BERT: bidirectional Vision Transformer ViT: image patches Mixture of experts routed FFN experts Diffusion U-Net conv + attention HOW EVERY ONE IS TRAINED The training loop loss → backward → clip → AdamW step → LR schedule, under autocast Solid arrows: "is built from". Dashed lines: "is trained by".

Shared vocabulary

Every page uses the same names. A model reads a batch of token IDs (integers that index a vocabulary), looks each one up in an embedding table to get a vector of width d_model, passes those vectors through N layers, and ends with one score per vocabulary entry, the logits. Inside a layer, attention splits d_model into heads slices of width d_head = d_model / heads.

ComponentShape/ParamsMeaningSource
Token IDs [batch, seq], integers Which vocabulary entry sits at each position. seq is the sequence length. input to nn.Embedding
Embedding weight [vocab, d_model]; out [batch, seq, d_model] A lookup table: row i is the learned vector for token i. Embedding.weight
Linear weight [out, in] + bias [out] y = x·Wᵀ + b over the last dimension. Most of a transformer's parameters live in these. nn.Linear
Multi-head attention in_proj_weight [3·d_model, d_model] + out_proj: 4·d_model² + 4·d_model with bias Mixes information between positions. The packed weight holds the query, key and value projections; head_dim = embed_dim // num_heads. MultiheadAttention.__init__
LayerNorm weight + bias, each [d_model]: 2·d_model Rescales each position's vector to zero mean and unit variance, then applies a learned scale and shift. LayerNorm.__init__
Logits [batch, seq, vocab] Unnormalized scores for the next token (or the masked token) at each position; softmax turns them into probabilities. input to CrossEntropyLoss

PyTorch's own transformer modules default to batch_first=False, so nn.Transformer expects [seq, batch, d_model] unless you pass batch_first=True (constructor defaults). These pages write shapes batch-first, and say so whenever the PyTorch default matters.

Pinned releases

Code links go to one release of each library, so a line range keeps pointing at the right code after the library moves on. When a newer stable release changes a linked file, the pages and the pin move together. Formulas, results and model configurations that come from papers rather than library code cite the paper.

LibraryReleaseCommitUsed for
PyTorch v2.15.0 31a77fc9c0a7 Tensors, neural networks, training and inference
scikit-learn 1.9.1 866c0f51e756 Classical models, preprocessing, splits and metrics
XGBoost v3.4.2 fdf0888beddd Gradient-boosted trees

Pages are static. The few figures computed with a library are produced offline by scripts in this site's repository and committed as small data files; nothing here runs a model.

Pages by area

Mathematical foundations

Vectors, matrices, dot products, gradients, the chain rule, probability, distributions, expectation, Bayes' rule, entropy.

Core ML concepts

Supervised, unsupervised and self-supervised learning; losses; generalization; bias–variance; overfitting; regularization.

Data and evaluation

Splits, leakage, preprocessing, missing data, imbalance, cross-validation, metrics, calibration, uncertainty.

  • Train, validation and test splits intro

    Train, validation and test splits, and when a random split gives the wrong answer.

  • Data leakage core

    How information leaks from test to train, with concrete cases and a checklist to prevent it.

  • Preprocessing and feature scaling intro

    Scaling, encoding and transforming features, and which models care.

  • Missing data core

    Why values go missing, the imputation strategies, and how boosted trees handle missing values natively.

  • Class imbalance core

    Why accuracy misleads on rare classes, and the weighting, resampling and threshold fixes.

  • Cross-validation core

    k-fold and its stratified, grouped and time-series variants, and nested CV for model selection.

  • Classification metrics intro

    The confusion matrix, precision, recall, F1, ROC and PR curves, and choosing a threshold.

  • Regression metrics intro

    MSE, MAE, R² and friends, and how to match the metric to the decision.

  • Calibration core

    When a predicted probability means what it says, and how to fix it when it doesn't.

  • Uncertainty advanced

    Aleatoric and epistemic uncertainty, ensembles, and conformal prediction with guaranteed coverage.

Classical ML

Linear and logistic regression, decision trees, random forests, gradient boosting, SVMs, nearest neighbors, clustering, PCA.

  • Linear regression intro

    Least squares from the normal equations and from gradient descent, plus ridge and lasso.

  • Logistic regression intro

    Log-odds, the sigmoid and the cross-entropy loss: classification as one linear layer.

  • Decision trees intro

    Recursive splitting by Gini or entropy, worked on a toy dataset, and how to stop a tree overfitting.

  • Random forests core

    Bagging and feature subsampling: many decorrelated trees averaged to cut variance.

  • Gradient boosting core

    Boosting as gradient descent in function space, and XGBoost's second-order split gain, derived.

  • Support vector machines core

    The maximum-margin classifier, the hinge loss, and kernels.

  • Nearest neighbors intro

    Predicting from the closest training points, distance metrics, and the curse of dimensionality.

  • Clustering intro

    k-means, DBSCAN and Gaussian mixtures, and how to judge a clustering.

  • Principal component analysis core

    Principal components via the SVD, explained variance, and nonlinear alternatives.

Neural network fundamentals

Layers, activations, backprop, computation graphs, initialization, normalization, optimizers, LR schedules, the training loop.

Model architectures

CNNs, RNNs/LSTMs, Transformers, autoencoders, GNNs, mixture of experts, state-space models, vision transformers.

  • Convolutional networks core

    Convolution as a sliding dot product, output-size arithmetic, pooling, and residual CNNs.

  • RNNs and LSTMs core

    Recurrence, backprop through time, and the gates that let LSTMs remember.

  • The Transformer core

    The full encoder-decoder from Attention Is All You Need, block by block, and how nn.Transformer implements it.

  • Autoencoders core

    Compress, then reconstruct: bottlenecks, denoising, and the latent space.

  • Graph neural networks advanced

    Message passing over graphs: GCN, GraphSAGE and GAT.

  • Vision Transformer core

    ViT: images as sequences of patches, patch embedding as a strided convolution, and the class token.

  • Mixture of experts advanced

    Routing each token to a few of many feed-forward experts: top-k gating, load balancing, total versus active parameters.

  • State-space models advanced

    S4 and Mamba: recurrences that train like convolutions and run in linear time.

Generative modeling

Autoregressive models, VAEs, GANs, diffusion, flow matching.

  • Autoregressive models core

    Generating one piece at a time by the chain rule of probability, trained with teacher forcing.

  • Variational autoencoders core

    The ELBO, derived, and the reparameterization trick that makes it trainable.

  • Generative adversarial networks core

    A generator and a discriminator trained against each other, and how to keep the game stable.

  • Diffusion models advanced

    Learning to undo noise: the forward process, the ε-prediction objective, and DDPM and DDIM sampling.

  • Diffusion U-Net advanced

    Denoising diffusion and the U-Net that predicts the noise: down and up paths, skips, timestep embeddings and attention.

  • Normalizing flows and flow matching advanced

    Normalizing flows, continuous flows, and the flow-matching objective that trains them simply.

Language models

Tokenization, embeddings, positional encoding, attention, encoder and decoder LMs, pretraining, fine-tuning, preference optimization, decoding, KV caching.

  • Tokenization intro

    How text becomes tokens: byte-pair encoding worked on a toy corpus, and why tokenization matters.

  • Embeddings intro

    Embedding tables as lookups, the geometry they learn, and contextual versus static embeddings.

  • Positional encoding core

    How a model that sees a set learns order: sinusoidal, learned and rotary (RoPE) positions.

  • Attention core

    Scaled dot-product and multi-head attention with a tiny numeric example, masks, and cross-attention.

  • Encoder-only models core

    The BERT-style stack: bidirectional attention, padding masks, masked-language-model training and task heads.

  • Decoder-only LLMs core

    The GPT-style stack: causal self-attention blocks, the LM head, generation, and a GPT-2-small parameter count.

  • Pretraining and scaling laws advanced

    Pretraining objectives, compute budgets, and the scaling laws that size a model.

  • Fine-tuning and LoRA core

    Full fine-tuning, instruction tuning, and LoRA's low-rank update, derived.

  • Preference optimization: RLHF and DPO advanced

    Reward models, RLHF with PPO, and DPO's closed-form loss.

  • Decoding and sampling core

    Greedy, beam, temperature, top-k and top-p sampling, and speculative decoding.

  • The KV cache advanced

    Why generation caches keys and values, how big the cache gets, and how MQA and GQA shrink it.

Retrieval and recommendation

Similarity search, vector indexes, ranking, collaborative filtering, two-tower models, RAG.

  • Similarity search intro

    Exact nearest-neighbor search over embeddings, the choice of metric, and lexical baselines.

  • Vector indexes advanced

    Approximate nearest neighbors with IVF, product quantization and HNSW.

  • Learning to rank core

    Pointwise, pairwise and listwise ranking, NDCG, and LambdaMART.

  • Collaborative filtering core

    Recommending from the user–item matrix: neighborhoods and matrix factorization.

  • Two-tower models core

    Separate query and item encoders trained with in-batch negatives, served with a vector index.

  • Retrieval-augmented generation advanced

    Retrieving passages and putting them in the prompt, and how to evaluate each half.

Training systems

GPU execution, precision, batching, distributed training, gradient accumulation, checkpointing, profiling.

  • How a GPU runs a model core

    Kernels, streams, memory bandwidth and the roofline: why some ops are fast and others aren't.

  • Precision and mixed precision core

    fp32, bf16, fp16 and fp8, autocast, and loss scaling.

  • Data loading and batching intro

    Datasets, DataLoaders, collation, padding and packing.

  • Gradient accumulation core

    Simulating a large batch with micro-batches, and the details that make it exact.

  • Distributed training advanced

    Data, sharded, tensor and pipeline parallelism, and memory per GPU worked through.

  • Checkpointing core

    Saving and resuming every piece of training state, and activation checkpointing.

  • Profiling core

    Measuring training correctly, reading a profiler trace, and computing MFU.

Inference and deployment

Quantization, distillation, serving, latency and throughput, monitoring and drift, reproducibility.

  • Quantization core

    Scale and zero point, per-channel quantization, and int8 and int4 for LLMs.

  • Knowledge distillation core

    Training a small student on a large teacher's soft targets.

  • Model serving core

    Exporting a model and serving it with batching on CPU or GPU.

  • Latency and throughput core

    Time to first token, tail latency, batching and Little's law.

  • Monitoring and drift core

    Detecting drift in inputs, predictions and outcomes after deployment.

  • Reproducibility intro

    Seeds, deterministic kernels and versioning: getting the same result twice.

Specialized learning

Reinforcement learning, time series, causal inference, online learning, multimodal models.

  • Reinforcement learning core

    MDPs, Q-learning, policy gradients and PPO.

  • Time series core

    Trend, seasonality, lag features, time-ordered validation and forecasting models.

  • Causal inference advanced

    Confounding, potential outcomes, propensity scores and experiments.

  • Online learning core

    Learning from a stream, regret, and bandits.

  • Multimodal models advanced

    Contrastive image–text pretraining and vision-language models.

Upcoming models

The models an ML engineer should know, from the classics to today's state of the art, oldest first within each tier. Each one has a single-sentence idea and its defining paper. Those with a page here link to it; more pages will follow. Most of the newest models are a Transformer variant, which is why that page comes first.

Classic machine learning

Small, fast, interpretable. Still the right first answer for most tabular data.

  • Linear regression 1805

    Fit y = Xw + b by least squares; the closed form and gradient descent agree.

    Legendre, 1805 (least squares)

  • Logistic regression 1958

    A linear score through a sigmoid, trained with cross-entropy: a one-layer neural network.

    Cox, The Regression Analysis of Binary Sequences, 1958

  • Naive Bayes 1960s

    Bayes' rule with features assumed independent given the class; a strong text baseline.

    Maron, Automatic Indexing, 1961

  • k-nearest neighbours 1967

    Predict from the k closest training points; no training, all cost at query time.

    Cover and Hart, Nearest Neighbor Pattern Classification, 1967

  • k-means and PCA 1901/1982

    Unsupervised workhorses: cluster by nearest centroid; project onto directions of most variance.

    Lloyd, Least Squares Quantization in PCM, 1982; Pearson, 1901

  • Support vector machines 1995

    The maximum-margin separator, with kernels for non-linear boundaries.

    Cortes and Vapnik, Support-Vector Networks, 1995

  • Decision trees and random forests 1984/2001

    Recursive splits on features; average many decorrelated trees to cut variance.

    Breiman, Random Forests, 2001

  • Gradient-boosted trees (XGBoost, LightGBM) 2016

    Add trees one at a time, each fitting the previous ensemble's gradient. Still state of the art on tabular data.

    Chen and Guestrin, XGBoost, 2016

Deep learning foundations

The building blocks every modern model is assembled from.

The transformer era

Attention replaces recurrence, and scale becomes the main lever.

Current state of the art

Open-weight models with published architectures. Closed frontier models don't publish theirs, so these pages don't guess at them.