Supervised, unsupervised and self-supervised learning

Problem

A model improves by being scored against something, so training needs a signal. The expensive part is rarely the model; it is the signal. Labelling a million photos with the objects in them takes people and money, while a trillion words of raw text cost almost nothing. Which setup you are in decides how much data you can use, what the model can learn, and what the loss measures (see loss functions).

Confusing the setups causes real mistakes: expecting clustering to recover the categories you had in mind, treating a pretraining loss as a quality score, or paying for labels a free objective could have supplied.

Intuition

Think of studying for an exam. Supervised: you have worked problems with answer keys. Unsupervised: you have a pile of problems with no answers and sort them into piles that look alike. Self-supervised: you cover a word in a textbook sentence and try to recall it; the book itself is the answer key, so you can quiz yourself for free, endlessly.

The only difference between the three is the source of the target.

Mechanics

Supervised learning

You have pairs (x_i, y_i) and fit a function f so that f(x_i) ≈ y_i by minimizing a loss. If y is a number it is regression (linear regression); if y is a category it is classification (logistic regression). The label defines the task: change it and the same inputs teach something else.

Unsupervised learning

You have only x. The goal is structure: groups (clustering), a compact description (dimensionality reduction), or the density p(x) itself. k-means is the standard example. It alternates two steps until nothing changes: assign each point to its nearest centre, then move each centre to the mean of its points. The quantity it lowers is inertia, the sum of squared distances from each point to its centre. Neither step can raise it, so it settles, though possibly in a poor spot that depends on the start. Nothing says the groups it finds match any human categories.

Self-supervised learning

You still only have x, but you manufacture a supervised problem from it by hiding a piece and predicting it. For text there are two standard choices:

• Next-token prediction: the input is tokens up to position t and the target is token t+1. A sequence of n tokens yields n − 1 labelled examples. This is how a decoder-only model is pretrained, and what decoding later samples from.
• Masked prediction: replace some tokens with a mask symbol and predict the originals from the surrounding words on both sides. The loss is a cross-entropy at the masked positions only.

The loss is the same cross-entropy as in supervised classification; only the labels' origin differs. For images, audio and video the hidden piece is a patch, a frame or the other view of the same item (the contrastive losses on the loss functions page).

In between

Semi-supervised learning has a few labels and many unlabelled points, and uses the unlabelled ones to learn the shape of the data, as in the worked example below. Weak supervision uses labels that are cheap but noisy: keywords, rules, links, or another model's guesses. Pretrain-then-fine-tune chains self-supervised pretraining on lots of raw data with supervised training on a small labelled set. Reinforcement learning is another case with its own signal, a reward, not covered here.

The estimator API

scikit-learn expresses the setups through one interface. Every estimator has fit(X, y=None); supervised ones need y, unsupervised ones ignore it. Then predict(X) for a supervised model, or labels_ and predict for a clusterer. Mixins add behaviour by role: ClassifierMixin.score is accuracy, RegressorMixin.score is R², and ClusterMixin adds fit_predict. PyTorch has no such interface; you write the loop yourself (see the training loop).

Feature 2 is the vertical axis. Colour shows the cluster k-means found; the two circled points are the only ones with labels.

Worked example

Implementation

import torch
import torch.nn.functional as F
from sklearn.cluster import KMeans
from sklearn.linear_model import LogisticRegression

# supervised: the labels y come from people
clf = LogisticRegression().fit(X, y)
clf.score(X_test, y_test)                       # accuracy

# unsupervised: no y; the labels_ are arbitrary cluster ids
km = KMeans(n_clusters=2, n_init=10).fit(X)
km.labels_, km.inertia_

# self-supervised: the targets are the inputs shifted by one
def next_token_loss(model, tokens):             # tokens: [B, T] ints
    logits = model(tokens[:, :-1])              # [B, T-1, V]
    return F.cross_entropy(
        logits.reshape(-1, logits.size(-1)), tokens[:, 1:].reshape(-1)
    )

# masked prediction: loss only where a token was hidden
def masked_loss(model, tokens, mask):           # mask: [B, T] bool
    corrupted = tokens.masked_fill(mask, MASK_ID)
    logits = model(corrupted)
    targets = tokens.masked_fill(~mask, -100)   # ignore_index default
    return F.cross_entropy(logits.transpose(1, 2), targets)

Note that KMeans.fit(X) takes no y, whereas LogisticRegression.fit(X, y) requires it, which is the whole distinction in one line. The self-supervised loss is the same cross_entropy call as the supervised one; -100 is its default ignore_index, which is what lets unmasked positions contribute nothing.

Tradeoffs

• Cost of labels. Supervised data is limited by annotation; self-supervised data is limited only by what you can collect. The price is that the objective is a proxy: predicting words is not the same as being helpful or correct.
• Evaluation. Supervised models have an obvious test, accuracy on held-out labels. Unsupervised ones have no ground truth, so you judge by indirect measures (inertia, downstream use) that can be gamed or misleading.
• Label quality. A supervised model learns the labellers' mistakes and biases. Weak labels scale but add noise the model can fit.
• k-means pitfalls. You must choose k; it assumes roughly round, similar-sized groups; and different starts give different results, which is why n_init exists. Lower inertia with larger k is not evidence of better clusters, since k = n gives zero.
• Mixing setups. Pretraining then fine-tuning gets the best of both but inherits the pretraining data's biases, and a fine-tuned model can forget what it learned in pretraining if tuned too hard.

Check yourself

Connections

References

• Hastie, Tibshirani and Friedman. The Elements of Statistical Learning, 2nd ed., chapters 2 and 14. Springer, 2009.
• Lloyd. Least Squares Quantization in PCM. IEEE Transactions on Information Theory, 1982 (circulated as a Bell Labs memo in 1957).
• Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018 (arXiv 1810.04805), for masked prediction.
• Radford et al. Language Models are Unsupervised Multitask Learners (GPT-2). OpenAI, 2019, for next-token pretraining.
• scikit-learn 1.9.1: BaseEstimator, KMeans.fit.
• PyTorch v2.15.0: CrossEntropyLoss.