Chapter 2 of 7 · 5 practices · 3 min · 4-question quiz

Data splits and leakage

Leakage makes a weak model look strong. These practices, mostly from scikit-learn's common pitfalls guide, keep information from the evaluation data out of training.

Start: Split the data before any preprocessing, and fit transforms on the training set only

5 practices in this chapter

1Split the data before any preprocessing, and fit transforms on the training set onlyscikit-learn defines data leakage as using information that would not be available at prediction time when building the model, which gives overly optimistic performance estimates. Its rule is simple…
2Put preprocessing and the model in one PipelineA Pipeline chains transforms with the estimator, so every fit learns preprocessing from the training part only, and every predict applies it consistently. scikit-learn's docs recommend it for…
3Split by group or by time when rows are not independentRandom splits assume rows are independent and identically distributed. scikit-learn's cross-validation guide warns that this rarely holds in practice, and names two common breaks: For groups, the…
4Remove duplicates across splits and keep the test set for a final checkGoogle's ML Crash Course recommends three sets: training, validation for decisions during development, and a test set held back for the end. It gives two warnings: Duplicates inflate scores. Remove…
5Check every feature for values that are only known after the outcomeLeakage is not only a splitting bug. A feature can carry the answer: a refund_issued column in a model that predicts complaints; an account_closed_date in a churn model; a field filled in by the…

Check yourself on this chapter

4 questions, right here once you are signed in. Your progress is saved as you read.