Chapter 2 · Practice 5 of 5
Check every feature for values that are only known after the outcome
Why it matters
Leakage is not only a splitting bug. A feature can carry the answer:
- a
refund_issuedcolumn in a model that predicts complaints; - an
account_closed_datein a churn model; - a field filled in by the team that handles the outcome.
A quick test: if one feature alone gives a near-perfect score, assume leakage until proven otherwise. Ask when and by whom each column is written, and compare that with the prediction moment you wrote down when framing the problem. Rule #31 adds a quieter version: a table joined at training time may have changed since, so use the values as they were at prediction time.