Preprocessing and feature scaling

Problem

A learner sees an array of numbers and treats them as equally meaningful. It cannot know that one column is in years and another in dollars, so a feature measured in large units can dominate anything that adds, compares or takes steps in proportion to raw size: a distance, a dot product, a gradient. And words such as "red" or "blue" are not numbers at all; encoding them badly (as 0, 1, 2) invents an order that is not there.

Preprocessing is the set of simple, mostly invertible transformations that put features into a form a model can use. They are cheap and easy to get subtly wrong, usually by letting information from the test data reach the training step.

Intuition

Imagine measuring how far apart two houses are on a map whose east–west axis is marked in meters and whose north–south axis is marked in kilometers. If you ignore the labels and subtract numbers, the east–west gap, being a number in the thousands, swamps the north–south gap, even when the houses are equally far apart in both directions. Scaling redraws the map so a step of "typical size" along either axis counts the same.

Encoding is the opposite problem: giving a label a position in space without implying a distance that does not exist. Red is not "one less than green"; one-hot encoding puts every category on its own axis, equally far from the others.

The same two numbers, computed once on the training rows, are used for every split. Test rows never contribute to them.

Mechanics

Scaling numeric features

Standardization (z-score) subtracts the column mean μ and divides by the standard deviation σ: z = (x − μ) / σ. The result has mean 0 and spread 1, and keeps the shape of the distribution. Min-max scaling maps the observed minimum to 0 and the maximum to 1: x′ = (x − min) / (max − min). It bounds the range but is moved around entirely by a single extreme value. Robust scaling uses the median and the interquartile range instead of the mean and standard deviation, so outliers have little say. Max-abs scaling divides by the largest absolute value and keeps zeros as zeros, which suits sparse data.

A constant column has zero spread; scikit-learn treats a zero scale as 1 rather than dividing by zero.

Who cares about scale

• Gradient descent. With features on very different scales the loss surface is a long narrow valley: one learning rate is too large for the steep direction and too small for the shallow one, so training zigzags and crawls. Standardized inputs give rounder bowls and faster convergence.
• Distance-based models (k-nearest neighbors, k-means, SVMs with an RBF kernel). The distance is a sum of squared differences, so the largest-unit column decides it.
• Regularized linear models. Ridge and lasso penalize the size of coefficients, and a coefficient's size depends on the feature's units.
• PCA. It finds the directions of greatest variance, which means the biggest-unit feature unless you standardize.
• Trees and tree ensembles do not care. A split asks "is x > t?", and any monotone rescaling of x moves t without changing which rows fall on each side.

Encoding categories

• One-hot creates one 0/1 column per category. No order is implied. Cost: width grows with the number of categories, which hurts for thousands of them.
• Ordinal maps categories to integers 0, 1, 2…. It is right only when the categories really are ordered (small, medium, large). Otherwise a linear model reads "blue = 0, red = 2" as a distance.
• Categories are sorted, so the column order is alphabetical or numeric, and a category that appears only at prediction time must be handled explicitly (scikit-learn's handle_unknown).

Skewed features and log transforms

Incomes, prices, counts and durations are often right-skewed: a few huge values stretch the mean away from the typical row. Taking a logarithm compresses the tail, turns multiplicative differences into additive ones, and makes the spread more even. PowerTransformer (Box-Cox and Yeo-Johnson) learns the power that best makes a column look Gaussian; for counts that include zero use log(1 + x).

fit versus transform

Every scikit-learn transformer has two steps. fit(X) learns the parameters from data (the mean and standard deviation, the category list, the min and max) and stores them as attributes ending in an underscore. transform(X) applies those stored parameters to any rows. fit_transform does both on one array. The rule is: fit on the training rows only, then transform train, validation and test with that same fitted object. Refitting on the test rows silently gives the test set a different ruler.

Same five people. In raw units the x differences are small compared with the y differences, so a distance sees almost only income.

Worked example

Implementation

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

pre = ColumnTransformer([
    ("num", StandardScaler(), ["age", "income"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["color"]),
])
model = Pipeline([("pre", pre), ("clf", LogisticRegression())])

model.fit(X_train, y_train)        # scaler and encoder learn from train only
model.score(X_test, y_test)        # test rows use the stored mean_, scale_, categories_

# by hand, the same rule:
scaler = StandardScaler().fit(X_train)       # learn mean_ and scale_
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)          # same ruler, never fit on test

Putting the steps in a Pipeline makes the rule automatic: when you cross-validate it, the scaler is refit inside each training fold and never sees the validation fold.

Tradeoffs

• Standardization or min-max? Standardization tolerates a few large values better and is the usual default. Min-max is handy when you need a bounded range (pixel intensities, inputs to some neural nets) but one outlier compresses everything else.
• Outliers. Mean and standard deviation are themselves pulled by outliers; robust scaling or a log transform is safer when the column is heavy-tailed.
• Interpretability. Scaled coefficients are per standard deviation, not per original unit; keep the fitted scaler to translate back.
• One-hot width. A column with 10,000 categories becomes 10,000 columns. Sparse output, grouping rare categories or learned embeddings are the usual escapes.
• Trees do not need scaling. Skipping it for them saves work, but a pipeline with a mix of model types often scales everything for uniformity; it is harmless.
• Sparse data. Centering turns zeros into non-zeros and destroys sparsity; use with_mean=False or max-abs scaling.
• Leakage. Scaling is the most common way test information gets into training: compute the statistics on all rows before splitting and the model has quietly seen the test set. See data splits for the splitting side of the same discipline.

Check yourself

Connections

References

• Hastie, Tibshirani and Friedman. The Elements of Statistical Learning, 2nd ed., section 3.4 (ridge regression and standardized inputs). Springer, 2009.
• LeCun, Bottou, Orr and Müller. Efficient BackProp. In Neural Networks: Tricks of the Trade, 1998, on centering and normalizing inputs.
• Box and Cox. An Analysis of Transformations. Journal of the Royal Statistical Society B, 1964; Yeo and Johnson, A New Family of Power Transformations to Improve Normality or Symmetry. Biometrika, 2000.
• scikit-learn 1.9.1: StandardScaler, MinMaxScaler, OneHotEncoder, OrdinalEncoder.