Sefism MLflow Cheat Sheet

What this cheat sheet is

The Sefism MLflow cheat sheet is a quick-revision book on MLflow in Python for engineers who train, compare, ship and monitor models. It follows a model's life: tracking experiments, packaging models, registering and promoting them with aliases, serving and evaluating them, tracing GenAI apps, and the habits that make machine learning work reproducible.

It is a standalone sheet with no editions: nine chapters and 65 concepts, the same for every reader. Each concept is a short card with what it is, why it matters and, for many, the mistake people make with it, often with the function or command you would actually type. Where MLflow 3 changed things, such as log_model taking a name and logged models becoming first-class objects, the cards say so.

Who it suits

• Data scientists and ML engineers who already train models and want runs they can compare and reproduce.
• Engineers setting up a shared tracking server and registry for a team.
• Developers building LLM applications who want tracing, prompt versioning and evaluation in one place.

If you are earlier on, the Machine learning roadmap teaches the modelling itself. If you want the full path from notebook to production, the MLOps and LLMOps roadmap covers pipelines, deployment and monitoring in order, and its own MLOps cheat sheet covers the wider practice. This sheet goes deeper on one tool.

Chapter by chapter

1. Tracking experiments

What MLflow is, experiments and runs, parameters (written once per run and stored as strings), metrics with their full history and step, tags, artefacts, dataset tracking with mlflow.data and log_input, and system metrics for CPU, memory, disk and GPU use. The first pitfall is the most common one: logging everything to the Default experiment.

2. Autologging, search and the UI

mlflow.autolog() and the library-specific autolog functions, nested runs for hyperparameter searches, mlflow.search_runs with its SQL-like filters, the lower-level MlflowClient, the MLflow UI with run comparison and parallel coordinates plots, and conventions for organising runs. A useful catch: parameters are stored as strings, so they do not compare as numbers in a filter.

3. Tracking server and stores

The tracking URI and MLFLOW_TRACKING_URI, local tracking, the tracking server, the backend store (PostgreSQL or MySQL for teams, SQLite for one person), the artefact store on object storage, artefact proxying, and authentication. The open-source server has no login by default; the card explains the basic-auth app option and the alternatives.

4. Models, flavours and signatures

The MLflow Model directory and its MLmodel file, flavours, log_model, logged models and model URIs in MLflow 3, model signatures and infer_signature, input examples, the generic pyfunc flavour, custom Python models that wrap preprocessing or thresholds, and model dependencies.

5. Model Registry

Registered models and versions, aliases such as champion and challenger, the deprecated stages they replace, version tags and descriptions, loading by alias with models:/<name>@<alias>, promotion across environments with copy_model_version, and lineage back to the run and data. One pitfall worth remembering: a running service keeps the version it loaded until it reloads, even after the alias moves.

6. Serving, deployment and projects

mlflow models serve and its /invocations endpoint, request formats, environment managers, building a container with mlflow models build-docker, batch scoring with pyfunc or spark_udf, managed deployment targets, and MLflow Projects with an MLproject file and mlflow run.

7. Evaluation

Evaluating classical models with mlflow.models.evaluate, custom business metrics, champion versus challenger on the same held-out data, evaluating GenAI apps with mlflow.genai.evaluate, scorers and LLM judges, and evaluation datasets that grow from real failures.

8. GenAI tracing and prompts

Traces as trees of spans, autolog for LLM libraries such as mlflow.openai.autolog(), manual spans with @mlflow.trace, feedback on traces, the Prompt Registry with versioned templates, prompt aliases, and versioning a GenAI app as a logged model in MLflow 3.

9. MLOps practice

Reproducible runs, training pipelines run by an orchestrator such as Airflow, Prefect or Dagster, promotion in CI by moving the champion alias only when thresholds pass, monitoring and drift in production, training and serving consistency, housekeeping with mlflow gc, and team conventions such as required tags and who may move production aliases.

Rules of thumb

The sheet closes with eight rules:

• Set MLFLOW_TRACKING_URI and an experiment explicitly in every environment, including CI.
• Log a signature and an input example with every model you might serve.
• Load models by alias, such as models:/churn@champion, and promote by moving the alias.
• Use aliases and tags instead of the deprecated registry stages in all new work.
• Evaluate challenger and champion on the same frozen dataset before any promotion.
• Record the dataset with log_input so every model can be traced to its training data.
• Never expose a tracking server without authentication in front of it.
• Turn on tracing in GenAI apps from day one and grow the evaluation set from bad traces.

How to use it

• Starting a project, read chapters 1 to 3 and set up tracking properly before your first real run.
• Before you ship a model, use chapters 4 to 6 as a checklist: signature, input example, dependencies, alias and serving format.
• Before a promotion, revise chapter 7 so the comparison is fair.
• Building an LLM app, go straight to chapters 7 and 8.
• Setting team rules, chapter 9 gives you the conventions to agree on.

How it works on Sefism

Everything on Sefism is free to learn from. The public page lists all nine chapters, with the first chapters open to read. Keeping the sheet in your member area uses credits, like any cheat sheet; because it is standalone, it is not included with a roadmap. New members get starting credits when they join. If you have a question about your own setup, send Tauseef a query.

Related on Sefism

• Roadmaps: MLOps and LLMOps, with its cheat sheet, and Machine learning
• Question: How do I become an MLOps engineer?
• FYP ideas: Pharmacy Demand Forecasting Service with Drift Monitoring and Automated Retraining and an evaluation harness for language model applications
• Resources: MLflow Documentation, MLOps Zoomcamp, Made With ML, Designing Machine Learning Systems

FAQ

What is MLflow used for? It is an open-source platform for the machine learning life cycle: tracking experiments, packaging models, a model registry, serving, evaluation, and tracing for GenAI apps.

What replaced model stages in the MLflow Model Registry? Aliases and tags. The fixed stages (None, Staging, Production, Archived) are deprecated; you point an alias such as champion at a version and load it with models:/<name>@champion.

Do I need a tracking server? Not to start: with no tracking URI set, MLflow records runs locally and mlflow ui browses them. A team needs a shared server with a database backend store, an artefact store and authentication.

Can MLflow track LLM applications? Yes. Tracing records each request as spans, the Prompt Registry versions prompts, and mlflow.genai.evaluate scores app outputs with code-based scorers and LLM judges.