Machine Learning Best Practices

Chapter 2 · Practice 1 of 5

Split the data before any preprocessing, and fit transforms on the training set only

From Data splits and leakage

Why it matters

scikit-learn defines data leakage as using information that would not be available at prediction time when building the model, which gives overly optimistic performance estimates. Its rule is simple: never call fit on the test data.

That includes preprocessing. A scaler, an imputer or a feature selector fitted on all the data has already seen the test rows. The docs show how large the effect can be: with 10,000 random features and random labels, selecting features before splitting gave an accuracy of 0.76, while selecting them on the training data only gave 0.5, which is what random labels deserve.

python
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
scaler = StandardScaler().fit(X_train)   # learn from train only
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)      # apply, never refit

Avoid

  • Calling fit or fit_transform on the full dataset before splitting