Chapter 2 of 7 · 5 checks · 2 min

Features and training-serving consistency

A model only performs in production as well as the features it receives there. These checks keep training and serving features the same.

Show

Use the same feature code in training and servingMust
Why it matters and how to check
Rule #32 is to reuse code between the training pipeline and the serving pipeline whenever possible. Google Cloud's MLOps guidance lists training-serving skew as a key problem of manual ML processes, often caused by features being reimplemented for production. Two implementations of the same feature will eventually disagree.
To verify, trace each feature from the training job and from the serving path back to its code. They should call the same function or read from the same feature store definition, not two copies.
Log the features used at serving timeMust
Why it matters and how to check
Rule #29 says the best way to make sure you train like you serve is to save the features used at serving time and use them for training, at least for a sample. Logged serving features are also what you need to debug a bad prediction later.
To verify, pick a recent prediction id and retrieve the exact feature values the model received, the model version and the output. If you can only recompute features after the fact, you cannot prove what the model saw.
Snapshot or version lookup tables that features are joined fromShould
Why it matters and how to check
Rule #31 warns that if you join data from a table at training and serving time, the data in that table may change in between. A customer attribute updated after training makes historical training rows look different from what the model will see.
To verify, list every table joined into features. For each, confirm the training job reads a point-in-time snapshot or versioned copy that matches what serving would have seen at that time.
Validate input data schema and value ranges before training and at servingMust
Why it matters and how to check
Google Cloud's MLOps guidance describes data validation as a gate in the pipeline: schema skews, such as unexpected or missing features, should stop the pipeline, while value skews, such as shifted distributions, should trigger investigation or retraining.
To verify, feed the training pipeline a batch with a missing column and a batch with values outside the expected range. The first should halt with a clear error; the second should raise an alert. Do the same for a malformed request at the serving endpoint.
Give every feature an owner and a descriptionShould
Why it matters and how to check
Rule #11 is to give feature columns owners and documentation. When a feature's source changes or behaves oddly, someone has to know what it is supposed to mean and who decides how it changes.
To verify, check the feature list or feature store: every feature used by the production model should have an owner, a description, its source and its units.