Model Card and Datasheet Generator from Training Runs
Model cards and dataset datasheets are the standard way to document what a model does, what data it learned from and where it fails, but they are written by hand at the end, if at all. Build a generator that pulls facts from MLflow runs, dataset files and evaluation results, drafts a model card and datasheet, flags missing or unsupported sections, and keeps the card in sync as the model is retrained.
Beginner
Achievable by one student in a semester with familiar tools.1 semester · 1 to 2 students
7 to ship
3 optional extrasSuggested stack
What you should ship
- Collectors that read parameters, metrics, artefacts and lineage from MLflow runs, and profile a dataset (size, columns, class balance, missing values, languages)
- Templates for a model card and a datasheet based on the published formats, with required and optional sections
- Sliced evaluation: performance broken down by groups the user defines (for example language, region or gender where appropriate and lawful), included in the card
- A completeness checker that flags empty or vague sections such as intended use, out-of-scope use and known limitations
- Output to Markdown for Hugging Face Hub and to a PDF for reports
- A command that regenerates the card on each new run and shows a diff
- Cards generated for three of your own models trained on public datasets, reviewed by a teacher
If you have time left
- A GitHub Action that blocks a model release when required sections are missing
- A section mapping card fields to EU AI Act technical documentation headings
- Urdu-language summary for non-technical readers
The problem: models ship without documentation
A student team trains a model, posts it to Hugging Face and moves on. A company fine-tunes a classifier and deploys it. In both cases, a year later nobody remembers what data it saw, what it was tested on, or which groups it performs worse for. That is how models get reused for jobs they were never fit for.
"Model Cards for Model Reporting" (Mitchell and colleagues, 2019) and "Datasheets for Datasets" (Gebru and colleagues) proposed short, structured documents to fix this. Hugging Face made model cards the default README of every model repository. The EU AI Act also expects technical documentation for high-risk systems and general-purpose models. Yet most cards are half-empty templates.
What already exists and the gap
- Hugging Face model card templates and a card-writing guide.
- Google's Model Card Toolkit, an earlier open source project for generating cards from TensorFlow pipelines.
- MLflow records runs, parameters and metrics, but does not write documentation.
The gap is a generator that uses the facts you already logged, fills in what it can, refuses to invent what it cannot, and keeps the card current as the model changes.
Inspiration
The original model card paper argued for sliced evaluation: reporting results for different groups, not just one average. Automating that slice is the most valuable part of this project.
What you build
- Collectors for runs and datasets.
- Templates and a completeness checker.
- Sliced evaluation built into the card.
- Regeneration and diffs per training run.
The technically interesting part
- What can be automated. Metrics and data stats can be filled in. Intended use and limitations need a person. Your tool should prompt for them clearly and flag them when they are missing.
- Sliced metrics with small groups. Show confidence intervals so a slice with 12 examples is not over-interpreted.
- Staying in sync. Link each card to a run ID so an outdated card is obvious.
How you will evaluate it
- Completeness scores for cards before and after using the tool on three models.
- A review by a teacher or ML practitioner rating usefulness.
- Time saved compared with writing a card by hand.
Path to a product
Documentation is a recurring pain for ML teams and is becoming a compliance requirement. A plug-in for MLOps platforms or a hosted documentation service are realistic paths. It is also a gentle, practical way into AI governance work.
Scope warning
Use public datasets. Only slice by sensitive attributes when the dataset legitimately includes them and you can explain why. Do not use an LLM to invent limitations; let it suggest questions, and let a person answer.
Learn what you need on Sefism
- AI governance roadmap: documentation, transparency and why cards matter.
- MLflow cheat sheet: runs, artefacts and the model registry.
- MLflow documentation: the API your collectors use.
- Hugging Face Hub: where your cards will be published.
- Machine learning roadmap: evaluation and train-test practice behind the numbers.
- AI final year project if not good at math: why a tooling project like this is a strong choice.
- What does an AI governance job involve?: who reads model cards and why.
Get help with this idea
The risk is a generator that produces confident-looking but empty cards. Tauseef can help you decide what to automate, what to ask people, and how to show the tool improves real documentation. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.