Chapter 1 of 7 · 4 checks · 2 min
Baselines, metrics and scope
Decide what the model must beat and how you will measure it before you deploy anything. These checks set the bar the rest of the list measures against.
Have a simple baseline, such as a heuristic or the current system, to compare againstMust
Why it matters and how to check
Rule #1 of Google's Rules of ML is not to be afraid to launch without machine learning. A heuristic, a rule set or the current process is the baseline the model has to beat in production, and sometimes it is good enough to ship first.
To verify, find the baseline's score on the same evaluation data and metrics as the model. If nobody can say how much better the model is than the simple option, the deployment decision is not yet informed.
Define and instrument the product and model metrics before launchMust
Why it matters and how to check
Rule #2 is to design and implement metrics first. Decide which offline metric the model is judged on, which product metric it is meant to move, and which guardrail metrics must not get worse, such as latency or complaint rate. Instrument them before launch so you have a history to compare against.
To verify, open the dashboard and confirm each metric is being recorded today, for the current system, before the model goes live.
Ship a simple first model and get the pipeline and serving rightShould
Why it matters and how to check
Rule #4 says to keep the first model simple and get the infrastructure right: reliable data flow, evaluation and serving. Rule #5 says to test the infrastructure independently from the model, for example by checking that data reaches the trainer complete and that the model scores the same in training and serving.
To verify, confirm the end-to-end path, from raw data to a served prediction, has been exercised with the simple model, and that problems found there were infrastructure problems you can now rule out for later, more complex models.
Score the system against a production readiness rubricNice to have
Why it matters and how to check
The ML Test Score paper from Google researchers presents 28 specific tests and monitoring needs as a road map for production readiness and for paying down ML technical debt. Scoring your system against a rubric like it shows which gaps are known and accepted and which have been missed.
To verify, keep the completed rubric with the model's documentation, with each gap marked as planned, accepted with a reason, or fixed, and review it before each major release.