Best practices · Data and ML
Data Pipeline Best Practices
Opens with creditsBest practices for batch data pipelines, checked against the Apache Airflow and dbt documentation and Google's guidance on data validation for ML systems. It covers idempotent, re-runnable tasks, orchestrator hygiene, layered transformations, data tests and freshness checks, ownership and lineage, and monitoring for silent failures.
Written by Tauseef Fayyaz6 chapters, 23 practicesAbout 13 min22 quiz questionsReviewed 13 October 2026What you will learn
6 chapters · about 13 min
1Idempotent, re-runnable tasksEvery task will be retried, cleared or backfilled at some point. These practices make re-running a task produce the same result instead of duplicates or gaps.4 practices · 3 min
2Orchestration code hygieneThe orchestrator parses your pipeline files constantly and runs tasks on different machines. These practices keep DAG files fast to load and tasks independent of where they run.4 practices · 2 min
- 1Keep DAG files light: no queries, API calls or heavy imports at the top level
- 2Read variables and configuration inside tasks or through templates
- 3Pass small values between tasks and keep large data in shared storage
- 4Test that every DAG loads, and run full DAGs in staging before production
4 in this chapterOpen this chapter- Layered transformationsTransformations that happen in many places drift apart. These practices, from dbt's project structure guide, give each transformation one home.4 practices · 2 minOpens with credits
- Data tests and freshnessCode tests check your logic; data tests check what actually arrived. These practices catch bad and late data before anyone reads it.4 practices · 2 minOpens with credits
- Ownership, schema changes and lineagePipelines outlive the people who built them. These practices make it clear who owns each dataset, what it means and what breaks when it changes.3 practices · 2 minOpens with credits
- Monitoring and operationsA pipeline that fails loudly is easy to fix; one that quietly produces wrong data is not. These practices make problems visible and recoverable.4 practices · 2 minOpens with credits
About this guide
A data pipeline that works once is easy. A data pipeline that can be re-run after a failure without duplicating rows, backfilled for last March without surprises, and trusted by the people reading the dashboard on Monday morning is a different job. These data pipeline best practices are the habits that make the second kind.
The guide uses Apache Airflow for orchestration examples and dbt for transformation examples, because both publish detailed best practice documentation and both are widely used. The ideas carry over to other tools. Whether you use Dagster, Prefect, a cloud scheduler or cron, the same rules apply: tasks should be safe to repeat, orchestration code should stay light, transformations should happen once in a known place, and every table someone relies on should be tested.
The most expensive pipeline failures are the quiet ones. A job that crashes gets fixed the same day. A job that keeps running on a stale upstream table, or loads half the rows because a source changed shape, can feed wrong numbers into reports and models for weeks. Google's Rules of Machine Learning calls these silent failures and warns that they happen more in machine learning systems, which live on data, than in other kinds of software. Two chapters here, on tests and on monitoring, are about catching them.
Start with the first chapter. Idempotent tasks make every other practice easier, because a pipeline you can safely re-run is a pipeline you can safely fix.
Common questions
What does idempotent mean in a data pipeline?
An idempotent task gives the same final result no matter how many times it runs for the same input and time window. In practice, that means using upserts, merges or partition overwrites instead of plain inserts, and reading a fixed data window rather than the latest data.
What are dbt staging, intermediate and mart models?
Staging models clean each source table one-to-one with renaming, casting and light computations. Intermediate models prepare staging models to be joined. Marts combine them into wide tables for business entities such as customers or orders. Each transformation lives in one layer only.
Which data tests should every pipeline have?
At minimum, unique and not_null tests on every model's primary key, relationships tests on foreign keys, accepted_values on important categories, and freshness checks on sources. Add singular tests for business rules, such as payments never being negative.
What is a silent failure in a data pipeline?
A silent failure is when jobs succeed but the data is wrong or stale, for example because a source stopped updating or a change cut the number of rows. Google's Rules of ML warns about these. Catch them by monitoring row counts, null rates and freshness, not only job status.
Should I use catchup or backfill in Airflow?
In Airflow 3, catchup is off by default, so the scheduler runs only the latest interval. For reprocessing past periods, use an explicit backfill with a date range and limited concurrency, after confirming your tasks are idempotent and read their own data interval.
Sources (8)
Checked against these, last on 13 October 2026.
- Best Practices (Airflow 3.3.2), Apache Airflow docsairflow.apache.org/docs/apache-airflow/stable/best-practices.html
- Dag Runs, Apache Airflow docsairflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html
- How we structure our dbt projects, dbt docsdocs.getdbt.com/best-practices/how-we-structure/1-guide-overview
- Staging: preparing our atomic building blocks, dbt docsdocs.getdbt.com/best-practices/how-we-structure/2-staging
- Add data tests to your DAG, dbt docsdocs.getdbt.com/docs/build/data-tests
- freshness, dbt docsdocs.getdbt.com/reference/resource-properties/freshness
- Rules of Machine Learning: Best Practices for ML Engineering, Google for Developersdevelopers.google.com/machine-learning/guides/rules-of-ml
- MLOps: Continuous delivery and automation pipelines in machine learning, Google Clouddocs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
Related on Sefism
Picked from the other libraries by shared topics.
Roadmaps
See allCheat sheets
See allBest practices
See all- Data Analysis Best PracticesAn analyst's workflow in SQL, pandas and Polars: framing the question, reading data correctly, joins and grain, aggregation pitfalls, reproducible notebooks, checking results and sharing findings.
- Data Modeling Best PracticesDimensional data modeling for analytics: declaring the grain, fact and dimension tables, slowly changing dimensions, dbt marts, naming, tests, contracts and a semantic layer.
- Data Visualization Best PracticesHow to make charts that are clear, accessible and honest: choosing the chart, encoding, color, axes and scales, annotation, dashboards and avoiding misleading charts.
Checklists
See all- Data Quality ChecklistA data quality checklist for warehouse tables: ownership, freshness, volume, schema, nulls, uniqueness, referential checks, anomaly monitoring and data incidents, with how to verify each.
- Analytics Dashboard ChecklistAn analytics dashboard checklist for launch day: purpose and audience, metric definitions, data freshness, correctness, performance, access, accessibility, ownership and adoption.
- ML Model Deployment ChecklistAn ML model deployment checklist: baselines, training and serving consistency, validation, model registry, canary rollout, rollback, drift monitoring and ownership, with how to verify each.
Project structures
See all- dbt project project structureA dbt project that turns raw tables into tested, documented models, using the staging, intermediate and marts layers from dbt's own structure guide. It runs on DuckDB on your laptop, so you can learn analytics engineering for free, and grows into a governed project with checks, contracts, slim CI and a hosted docs site that moves to BigQuery or Snowflake unchanged.
- Apache Airflow project structureFor data engineers scheduling real pipelines with Airflow 3. It starts from
astro dev init(dags, include, plugins, tests and a Runtime Dockerfile that runs Airflow locally in Docker) and grows into dags grouped by business area, logic in a tested include package, asset-driven scheduling, data quality checks, alerts, CI that parses and tests every dag, and dag-only deploys. A self-hosted docker compose and Helm route is shown alongside. - GenAI RAG app (FastAPI) project structureFor developers turning a "chat with your documents" demo into an app people can rely on. A FastAPI backend where every layer has a home: ingestion workers, retrieval, prompts as versioned files, model clients with fallbacks and cost caps, guards, evals that gate CI, and tracing, so a wrong answer can be traced to the step that caused it.
Career questions
See all- What is the difference between a data analyst, a data scientist and a machine learning engineer?Analysts answer business questions with SQL, spreadsheets, dashboards and reports. Data scientists add statistics, experiments and predictive models. ML engineers build and run models inside production software. Titles vary by company, so read the work in the job advert, and pick by the kind of problem you enjoy, not the title that sounds most senior.
- How do I switch from data analyst to data engineer?You already have the hardest part: SQL, business context and knowing what good data looks like. Add Python for pipelines, data modelling, an orchestrator, a cloud warehouse, Git and testing. Build one end-to-end pipeline, then move internally by taking on engineering work in your current team before applying outside.
- Can I get a remote data engineering job from Africa?Yes, it is realistic, and data engineering is one of the skills global employers most often hire remotely from Africa. You need solid SQL and Python, one cloud platform, a pipeline tool and a warehouse, two or three end-to-end projects, and a search aimed at remote-friendly companies and talent platforms. Most remote hires have some experience, so a local first job helps.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com