Best practices · Data and ML
Machine Learning Best Practices
Opens with creditsBest practices for building and evaluating a machine learning model before it goes anywhere near production, checked against the scikit-learn and PyTorch documentation, Google's Rules of Machine Learning and ML Crash Course, MLflow and Fairlearn. It covers problem framing and baselines, data splits and leakage, features, training and tuning, metrics and error analysis, reproducibility and experiment tracking, and fairness checks.
Written by Tauseef Fayyaz7 chapters, 30 practicesAbout 18 min25 quiz questionsReviewed 13 October 2026What you will learn
7 chapters · about 18 min
1Problem framing and baselinesDecide what the model predicts, what decision it supports and what it has to beat before you train anything. These practices set the bar every later chapter measures against.4 practices · 3 min
- 1Write down the prediction, the decision it drives and the cost of each kind of mistake
- 2Check whether a heuristic already solves the problem
- 3Score a dummy baseline on the same split and metric as your model
- 4Start with a simple, interpretable model before reaching for deep learning
4 in this chapterOpen this chapter2Data splits and leakageLeakage makes a weak model look strong. These practices, mostly from scikit-learn's common pitfalls guide, keep information from the evaluation data out of training.5 practices · 3 min
- 1Split the data before any preprocessing, and fit transforms on the training set only
- 2Put preprocessing and the model in one Pipeline
- 3Split by group or by time when rows are not independent
- 4Remove duplicates across splits and keep the test set for a final check
- 5Check every feature for values that are only known after the outcome
5 in this chapterOpen this chapter- FeaturesGood features matter more than clever models, especially early on. These practices come from Google's Rules of ML and scikit-learn's preprocessing tools.4 practices · 2 minOpens with credits
- Training and tuningTuning is where leakage most often sneaks back in. These practices keep hyperparameter search honest and training runs comparable.4 practices · 3 minOpens with credits
- Evaluation metrics and error analysisA single headline number rarely tells you whether a model is good enough. These practices pick the right metric and look past the average.5 practices · 3 minOpens with credits
- Reproducibility and experiment trackingIf you cannot rerun an experiment, you cannot tell whether a change helped. These practices control randomness and record what each run did.4 practices · 2 minOpens with credits
- Fairness checksA model can perform well on average and still fail some groups of people. These practices follow Fairlearn's assessment guide.4 practices · 2 minOpens with credits
About this guide
Most machine learning projects that disappoint do not fail because the model was too simple. They fail because the evaluation was wrong. A scaler fitted on the whole dataset, a customer who appears in both the training and the test set, a test set that was looked at fifty times during tuning, or an accuracy score on data where 99 out of 100 rows belong to one class: each of these produces a number that looks great and means little. These machine learning best practices are about building a model whose numbers you can believe.
The guide follows the order of real work. It starts with framing the problem and setting a baseline, because a model is only good relative to something simpler. It then covers splitting data without leakage, building features, training and tuning, choosing metrics and reading errors, keeping experiments reproducible, and checking how the model behaves for different groups of people. Examples use scikit-learn for classic models and PyTorch for deep learning, with MLflow for tracking and Fairlearn for group metrics, but the ideas apply to any library.
This guide stops where deployment starts. Serving, canary rollouts, drift monitoring and retraining are covered in the ML model deployment checklist. Everything here happens before that: on your laptop, in a notebook or in a training pipeline, while the model is still cheap to change.
Start with the first chapter even if you are eager to train. Google's Rules of Machine Learning open with a reminder that a simple heuristic may get you most of the way, and that you should design your metrics first. A clear problem and a baseline make every later decision easier to judge.
Common questions
What is data leakage in machine learning?
Data leakage is using information during training that would not be available when the model makes real predictions, which makes evaluation scores look better than they are. Common causes are fitting preprocessing on the whole dataset before splitting, letting the same customer or patient appear in both training and test data, and features that are only known after the outcome.
How should I split data for a machine learning model?
Use a training set, a validation set or cross-validation for decisions, and a test set you use once at the end. Split by group when rows share an entity such as a patient or user, and by time when the model will predict the future. Remove duplicates across splits before you evaluate.
Which metric should I use for an imbalanced classification problem?
Not accuracy, since a model that always predicts the majority class can score very high. Use recall when missed positives are costly, precision when false alarms are costly, or F1, balanced accuracy or average precision when you need one summary. Compare against a dummy baseline on the same metric.
Why use a scikit-learn Pipeline?
A Pipeline chains preprocessing and the model so that transforms are learned from training data only and applied the same way at prediction time. It prevents leakage during cross-validation and hyperparameter search, and it means you save and load one object instead of several steps you might forget.
How do I make a PyTorch training run reproducible?
Seed torch, Python's random module and NumPy, turn off cuDNN benchmarking, enable deterministic algorithms, and give DataLoader a seeded generator and a worker_init_fn. Record the library versions too, because PyTorch does not promise identical results across releases or platforms.
How do I check a model for fairness?
Name the harms and groups first, then compute your metrics per group with a tool such as Fairlearn's MetricFrame and look at the gaps and the group sizes. Choose fairness criteria that fit the context, since common criteria can conflict, and record the decision.
Sources (8)
Checked against these, last on 13 October 2026.
- Common pitfalls and recommended practices, scikit-learn docsscikit-learn.org/stable/common_pitfalls.html
- Cross-validation: evaluating estimator performance, scikit-learn docsscikit-learn.org/stable/modules/cross_validation.html
- Metrics and scoring: quantifying the quality of predictions, scikit-learn docsscikit-learn.org/stable/modules/model_evaluation.html
- Reproducibility, PyTorch docsdocs.pytorch.org/docs/stable/notes/randomness.html
- Rules of Machine Learning: Best Practices for ML Engineering, Google for Developersdevelopers.google.com/machine-learning/guides/rules-of-ml
- Datasets, generalization, and overfitting: Dividing the original dataset, Google ML Crash Coursedevelopers.google.com/machine-learning/crash-course/overfitting/dividing-datasets
- MLflow Tracking, MLflow docsmlflow.org/docs/latest/ml/tracking/
- Performing a Fairness Assessment, Fairlearnfairlearn.org/main/user_guide/assessment/perform_fairness_assessment.html
Related on Sefism
Picked from the other libraries by shared topics.
Roadmaps
See allCheat sheets
See allChecklists
See allProject structures
See all- Machine learning project project structureFor students and analysts whose model works in a notebook and now needs to be reproducible, reviewable, served and watched. It starts from the Cookiecutter Data Science v2 layout (immutable raw data, numbered notebooks, a real Python package) and grows into versioned data with DVC, typed parameters, data validation with pandera, experiment tracking and a model registry with MLflow, a FastAPI prediction service, drift monitoring and, for a team, scheduled retraining, model cards, fairness checks and shadow deployment.
- MLOps pipeline project structureFor engineers who can train a model and now need the training to be a pipeline anyone can rerun, review and release. A uv workspace holds the pipeline package (prepare, validate, featurize, train, evaluate, register steps run by DVC), a small shared feature package, and a separate FastAPI prediction service that loads models from the MLflow registry by alias. It grows into CI that reproduces the pipeline on pull requests, scheduled retraining, approved promotion and monitoring.
Career questions
See all- My machine learning projects never leave a Jupyter notebook. How do I learn to deploy a model?Move the notebook code into a small Python package, save the whole preprocessing and model pipeline, put it behind a FastAPI endpoint in a Docker container and deploy it. Then add what makes it production: tracked runs, a registry, a test in CI, prediction logging and a drift check. Start with batch predictions if you can.
- What is the difference between a data analyst, a data scientist and a machine learning engineer?Analysts answer business questions with SQL, spreadsheets, dashboards and reports. Data scientists add statistics, experiments and predictive models. ML engineers build and run models inside production software. Titles vary by company, so read the work in the job advert, and pick by the kind of problem you enjoy, not the title that sounds most senior.
- Should I learn Python or R for data science?For most people aiming at industry jobs, learn Python first: it covers analysis, machine learning and production code, and most data job adverts ask for it. Choose R first if you are heading into statistics, academic research, public health or survey work, where it is strong. Either way, SQL comes too, and the concepts transfer.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com