MLOps and LLMOps cheat sheet (LLM apps)

1. 1From notebook to packageThe shape of the whole lifecycle, the tools that fill each step, and how to turn notebook code into a tested, locked, containerised Python project that anyone can run.8 concepts

   1. 1The MLOps lifecycleThe loop a production model lives in: data is collected and validated, a model is trained and evaluated, approved and deployed, served to users, monitored, and…
   2. 2Tool map per stageA reference of common open source tools for each step of the lifecycle.
   3. 3Project layout and pyproject.tomlA repository where notebook code becomes importable modules under src/<package>/, with dependencies, entry points and tool settings declared in one…
   4. 4Dependencies and lockfilesA lockfile records the exact version of every package, including indirect ones, that your project resolved to. uv, a fast Python package and project manager…
   5. 5Config and secretsSettings that change between runs or environments, such as paths, hyperparameters, model names and API keys, kept out of the code: parameters in a versioned…
   6. 6An LLM client moduleOne module that every feature calls instead of the provider SDK directly.
   7. 7Tests for ML codeFast tests with pytest for the deterministic parts: feature functions, data loading, schemas and output shapes, plus a smoke test that trains on a tiny sample…
   8. 8Container imageA packaged file system with your code, a Python runtime and locked dependencies, built from a Dockerfile, that runs the same way on a laptop, in CI and in the…

   8 in this chapterOpen this chapter
2. 2Data and model versioningGit for code, DVC for large data and model files, validation before training, and a frozen evaluation set, so any model can be traced back to the exact data that made it.6 concepts

   1. 1DVC (Data Version Control)An open source command-line tool that versions large files next to Git.
   2. 2DVC pipelinesStages declared in dvc.yaml, each with a command, its dependencies and its outputs.
   3. 3Data validationAutomatic checks on data before it is used: expected columns and types, value ranges, share of missing values, uniqueness of keys and row counts within a band.
   4. 4Frozen evaluation setA fixed, versioned set of examples with known correct answers that every candidate model or prompt is scored on, kept separate from training data and changed…
   5. 5LineageThe recorded chain from a prediction back to the model version, the training run, the code commit, the parameters and the data version that produced it.
   6. 6Versioning documents and indexesTreating the document snapshot, the chunking settings, the embedding model and the resulting vector index as one versioned artefact for a retrieval-augmented…

   6 in this chapterOpen this chapter
3. Experiment tracking and the registryRecording every run so results can be compared and reproduced, and a registry that names which model version is live. The standalone MLflow sheet goes deeper into the tool itself.7 conceptsOpens with credits
4. Pipelines and orchestrationBreaking training into steps, running them as a graph on a schedule or trigger, and making every step safe to retry and rerun.7 conceptsOpens with credits
5. Serving: batch, real time and GPUsChoosing how predictions reach users, wrapping a model in a validated API, packaging it as a container, and the latency, batching and scaling ideas that decide cost.8 conceptsOpens with credits
6. CI/CD for MLAutomated checks on code, data and model quality before anything ships, and release strategies that make a bad model cheap to catch and undo.8 conceptsOpens with credits
7. Monitoring, drift and retrainingWatching the service and the model separately, detecting when live data stops looking like training data, and deciding when and how to retrain.8 conceptsOpens with credits
8. LLMOps: prompts, evals, tracing, costRunning LLM features like production software: versioned prompts, eval sets and metrics, judges you can trust, traces of every call, caching, token budgets and defences against prompt injection.10 conceptsOpens with credits
9. Fine-tuning and serving open modelsWhen to fine-tune, how LoRA makes it affordable, shrinking models with quantisation, serving them efficiently, and the GPU memory and cost arithmetic behind every choice.8 conceptsOpens with credits
10. Capstone: a monitored model in productionThe documents and habits that turn a deployed model into a service someone can own: targets, a model card, a runbook, cost numbers and an honest write-up.5 conceptsOpens with credits
11. 11Glossary, A to ZEvery concept in the sheet and the smaller terms it uses, in one list.153 terms

Ship, serve and monitor models.