AI Incident Log and Postmortem Tracker for LLM Products
When an LLM feature misbehaves (a harmful answer, a leaked system prompt, a wrong refund, a runaway cost spike), most teams handle it in a chat thread and forget it. Build an incident tracker designed for AI failures: capture the exact prompt, context and model version, classify the failure, run a blameless postmortem, turn the case into a regression test, and track whether the fix held.
Beginner
Achievable by one student in a semester with familiar tools.1 semester · 1 to 2 students
6 to ship
3 optional extrasSuggested stack
What you should ship
- An incident form that captures prompt, retrieved context, model and prompt version, user impact and severity, with a one-click import from a Langfuse trace
- A failure taxonomy based on public lists (for example the OWASP Top 10 for LLM Applications) with clear definitions and examples
- A blameless postmortem template with timeline, root cause, fix and follow-ups
- Export of each incident as a regression test case for promptfoo, and a status that shows whether the test passes on the current version
- Dashboards: incidents by category, time to fix, repeat incidents
- A demo with at least 20 realistic incidents from a small LLM app you build
If you have time left
- Public incident summaries in the format used by the AI Incident Database
- Alerting from production signals such as cost spikes or groundedness drops
- A Slack bot for reporting incidents
The problem: AI failures are handled once and forgotten
Software teams have mature incident practice: a ticket, a timeline, a postmortem, a fix and a test. AI features break that habit. The failing case is a prompt and a context nobody saved, the model version changed since, and the "fix" is a prompt tweak with no test. Three months later the same failure comes back.
Public trackers show how varied AI failures are. The AI Incident Database and the OECD AI Incidents Monitor collect real-world AI harms. For high-risk systems, the EU AI Act also includes serious incident reporting duties for providers.
What already exists and the gap
- Incident tools such as PagerDuty, Jira Service Management and incident.io handle outages, not wrong answers.
- LLM observability tools such as Langfuse and LangSmith capture traces and user feedback.
- Public AI incident databases record notable harms across companies.
The gap is a small, team-level tracker that links an AI incident to its trace, classifies it with a shared vocabulary, and closes the loop with a regression test.
Inspiration
Google's SRE book popularised blameless postmortems. This project applies the same discipline to AI behaviour, where the "root cause" might be retrieval, a prompt, a model update or missing guardrails.
What you build
- Incident capture with trace import.
- A taxonomy and postmortem flow.
- Regression test export and pass/fail tracking.
- Dashboards for trends.
The technically interesting part
- Reproducibility. Store everything needed to replay the case: prompt, context, tool results, model and parameters.
- From incident to test. Turning a failure into an automated check needs a grader; start simple (contains, regex, JSON schema) and use an LLM judge only when needed.
- A shared vocabulary. A clear taxonomy makes trends visible. Test it by asking two people to classify the same incidents.
How you will evaluate it
- Agreement between two people classifying the same 20 incidents.
- Share of incidents successfully turned into passing regression tests after a fix.
- A short usability test with developers.
Path to a product
Teams running LLM features need AI-specific incident practice. A tracker that integrates with observability and eval tools could be sold as an add-on or offered as an open source project with a hosted version.
Scope warning
Use incidents from your own demo app. Keep the taxonomy short. Do not build general incident management; integrate with existing tools where possible.
Learn what you need on Sefism
- AI governance roadmap: incident response and accountability for AI systems.
- AI evals cheat sheet: turning failures into regression tests.
- Platform and SRE roadmap: incident practice and blameless postmortems.
- Langfuse documentation: importing traces.
- promptfoo: running the regression suite.
- PagerDuty incident response guide: incident roles and postmortem practice.
- What does an AI governance job involve?: the roles that use incident records like these.
Get help with this idea
The risk is a nice form nobody uses after the first week. Tauseef can help you design the taxonomy, the trace import and the test loop so the tool earns its place. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.