Chapter 1 of 8 · 4 checks · 2 min
Use case fit
Before anything else, confirm the feature is a good job for a language model, and that the cost of its mistakes is one your users and business can carry.
Write down the user problem and the non-AI baseline the feature must beatMust
Why it matters and how to check
AI features are often built because the technology is available, not because users need them. Write one paragraph on the problem, who has it and how they solve it today. Then name the simplest non-AI alternative, such as a search box, a template or a rules engine, and say why the AI version is better for users.
To verify, the launch ticket should link this note, and the product owner should be able to state the baseline and the measured improvement over it, for example task completion time or success rate in user testing.
Assess what happens when the AI is wrong, and design the feature around those stakesMust
Why it matters and how to check
Google's PAIR guidance on errors notes that risk is higher when users are novices or multitasking and when the system is less confident, and that health, safety, financial decisions and sensitive social contexts call for extra checks and explanation. NIST's Generative AI Profile names confabulation, confidently stated false content, as one of the risks generative AI introduces or makes worse.
To verify, list the three worst realistic wrong outputs for this feature and who is harmed by each. Each should have a mitigation in the design, such as human review, citations, a narrower scope or a refusal path, and the list should be attached to the launch ticket.
Define the success and guardrail metrics you will watch after launchMust
Why it matters and how to check
Decide before launch what success looks like in production, such as acceptance rate of suggestions, tasks completed or tickets deflected, and which guardrail metrics would make you roll back, such as complaint rate, flagged outputs, error rate, latency or cost per user.
To verify, each metric should have a definition, a data source that already exists, a target, and a rollback threshold. Open the dashboard before launch and confirm every metric shows data from staging or a dogfood group.
Name one owner who can pause or roll back the featureMust
Why it matters and how to check
NIST's AI Risk Management Framework puts governance first: roles and responsibilities for AI risk should be clear. In practice that means one named person who watches the feature after launch, owns its incidents and has the authority to switch it off without a meeting.
To verify, the owner and a backup are named in the launch ticket and in the on-call rota for the first two weeks, and both know where the kill switch is.