Machine Learning Best Practices

Chapter 2 · Practice 5 of 5

Check every feature for values that are only known after the outcome

From Data splits and leakage

Why it matters

Leakage is not only a splitting bug. A feature can carry the answer:

  • a refund_issued column in a model that predicts complaints;
  • an account_closed_date in a churn model;
  • a field filled in by the team that handles the outcome.

A quick test: if one feature alone gives a near-perfect score, assume leakage until proven otherwise. Ask when and by whom each column is written, and compare that with the prediction moment you wrote down when framing the problem. Rule #31 adds a quieter version: a table joined at training time may have changed since, so use the values as they were at prediction time.