Best practices · Data and ML
Data Analysis Best Practices
Opens with creditsBest practices for answering questions with data in SQL, pandas and Polars, checked against Google's Good Data Analysis guide and the pandas, Polars and PostgreSQL documentation. It covers framing the question, reading data correctly, joins and grain, aggregation pitfalls, reproducible notebooks, checking results and communicating findings.
Written by Tauseef Fayyaz7 chapters, 28 practicesAbout 18 min24 quiz questionsReviewed 13 October 2026What you will learn
7 chapters · about 18 min
1Frame the question firstAn analysis is only as useful as the question behind it. These practices pin down what you are measuring and why before you write a query.4 practices · 3 min
2Read the data correctlySummary numbers hide most problems. These practices get you looking at what a row means, how values are distributed and how your tools read types.4 practices · 3 min
- Joins without surprisesA join that changes the number of rows you expected is the most common silent error in analysis. These practices check cardinality, unmatched rows and nulls every time.4 practices · 3 minOpens with credits
- Aggregation pitfallsAggregates compress thousands of rows into one number, and the compression hides mistakes. These practices cover nulls, ratios, additivity and slicing.4 practices · 3 minOpens with credits
- Reproducible notebooks and codeIf you cannot rerun an analysis next month and get the same numbers, you cannot defend it. These practices make notebooks and queries repeatable.4 practices · 2 minOpens with credits
- Check your results before trusting themSurprising results are usually bugs. These practices, mostly from Google's Good Data Analysis guide, help you tell a finding from a mistake.4 practices · 2 minOpens with credits
- Communicate findingsAn analysis only helps if people understand it and trust it. These practices make the answer, its limits and its evidence clear.4 practices · 2 minOpens with credits
About this guide
Most wrong numbers in a report are not caused by hard statistics. They come from ordinary steps done carelessly: a join that quietly duplicated rows, an average of averages, a filter nobody wrote down, a notebook that only works when its cells run in a lucky order. These data analysis best practices are the habits that stop those mistakes before a chart reaches a meeting.
The guide follows an analyst's workflow from start to finish. It begins with the question, because an analysis that starts from a dataset or a favourite technique tends to answer something nobody asked. It then covers reading the data the way it was actually collected, joining tables without changing their grain, aggregating without the classic traps, keeping notebooks reproducible, checking results before trusting them, and writing up findings so people can act on them.
Examples use SQL with PostgreSQL 18, pandas 3.0 and Polars 2.0. Both pandas and Polars changed defaults in their latest major versions: pandas 3.0 turns on Copy-on-Write and a dedicated string type, and Polars 2.0 runs lazy queries on its streaming engine, which does not keep row order unless you ask. Where those changes affect everyday analysis, the guide says so.
Much of the process advice comes from Google's Good Data Analysis guide by Patrick Riley, which is worth reading in full. If you also build the pipelines and tables you analyse, the data pipeline and data modeling guides in this library pick up where this one stops.
Common questions
What are the main steps of a data analysis?
Start with a written question and the decision it informs. Then learn how the data was collected, validate it, describe what it shows, and only then judge what that means. Check results against past numbers and other sources, and write up the answer first, followed by evidence, method and caveats.
Why does my join return more rows than expected?
The key you joined on is not unique on one side, so each matching row is repeated. Check uniqueness before joining, pass validate="m:1" or "1:1" in pandas and Polars, and compare row counts before and after the join.
Why does SUM return null in SQL instead of zero?
In PostgreSQL and standard SQL, every aggregate except count returns null when no rows are selected. Wrap the result in coalesce(sum(x), 0) if an empty set should count as zero.
Should I use pandas or Polars for analysis?
Both work well. pandas has the larger ecosystem and, from version 3.0, Copy-on-Write and a dedicated string type by default. Polars is built around expressions and lazy queries, and from version 2.0 runs lazy queries on a streaming engine that does not keep row order unless asked. Pick one per project and learn its defaults.
How do I make a Jupyter notebook reproducible?
Put parameters such as date ranges and random seeds in one cell, use fixed dates rather than relative ones, pin library versions in a lock file, and restart the kernel and run all cells before sharing. Move reusable code into a module the notebook imports.
Sources (8)
Checked against these, last on 13 October 2026.
- Good Data Analysis, Google for Developersdevelopers.google.com/machine-learning/guides/good-data-analysis
- pandas.merge, pandas docspandas.pydata.org/docs/reference/api/pandas.merge.html
- What's new in 3.0.0, pandas docspandas.pydata.org/docs/whatsnew/v3.0.0.html
- polars.DataFrame.join, Polars docsdocs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.join.html
- Version 2 upgrade guide, Polars docsdocs.pola.rs/releases/upgrade/2/
- Aggregate Functions, PostgreSQL docswww.postgresql.org/docs/current/functions-aggregate.html
- Subquery Expressions, PostgreSQL docswww.postgresql.org/docs/current/functions-subquery.html
- Modeling Fact Tables in Warehouse, Microsoft Learnlearn.microsoft.com/en-us/fabric/data-warehouse/dimensional-modeling-fact-tables
Related on Sefism
Picked from the other libraries by shared topics.
Roadmaps
See allCheat sheets
See allBest practices
See all- Data Pipeline Best PracticesHow to build data pipelines you can re-run, test and trust: idempotent tasks, orchestrator hygiene, layered transformations, data tests, freshness, ownership and monitoring.
- Data Modeling Best PracticesDimensional data modeling for analytics: declaring the grain, fact and dimension tables, slowly changing dimensions, dbt marts, naming, tests, contracts and a semantic layer.
- Data Visualization Best PracticesHow to make charts that are clear, accessible and honest: choosing the chart, encoding, color, axes and scales, annotation, dashboards and avoiding misleading charts.
Checklists
See all- Data Quality ChecklistA data quality checklist for warehouse tables: ownership, freshness, volume, schema, nulls, uniqueness, referential checks, anomaly monitoring and data incidents, with how to verify each.
- Analytics Dashboard ChecklistAn analytics dashboard checklist for launch day: purpose and audience, metric definitions, data freshness, correctness, performance, access, accessibility, ownership and adoption.
- ML Model Deployment ChecklistAn ML model deployment checklist: baselines, training and serving consistency, validation, model registry, canary rollout, rollback, drift monitoring and ownership, with how to verify each.
Project structures
See all- dbt project project structureA dbt project that turns raw tables into tested, documented models, using the staging, intermediate and marts layers from dbt's own structure guide. It runs on DuckDB on your laptop, so you can learn analytics engineering for free, and grows into a governed project with checks, contracts, slim CI and a hosted docs site that moves to BigQuery or Snowflake unchanged.
- Apache Airflow project structureFor data engineers scheduling real pipelines with Airflow 3. It starts from
astro dev init(dags, include, plugins, tests and a Runtime Dockerfile that runs Airflow locally in Docker) and grows into dags grouped by business area, logic in a tested include package, asset-driven scheduling, data quality checks, alerts, CI that parses and tests every dag, and dag-only deploys. A self-hosted docker compose and Helm route is shown alongside. - GenAI RAG app (FastAPI) project structureFor developers turning a "chat with your documents" demo into an app people can rely on. A FastAPI backend where every layer has a home: ingestion workers, retrieval, prompts as versioned files, model clients with fallbacks and cost caps, guards, evals that gate CI, and tracing, so a wrong answer can be traced to the step that caused it.
Career questions
See all- What is the difference between a data analyst, a data scientist and a machine learning engineer?Analysts answer business questions with SQL, spreadsheets, dashboards and reports. Data scientists add statistics, experiments and predictive models. ML engineers build and run models inside production software. Titles vary by company, so read the work in the job advert, and pick by the kind of problem you enjoy, not the title that sounds most senior.
- How do I switch from data analyst to data engineer?You already have the hardest part: SQL, business context and knowing what good data looks like. Add Python for pipelines, data modelling, an orchestrator, a cloud warehouse, Git and testing. Build one end-to-end pipeline, then move internally by taking on engineering work in your current team before applying outside.
- Can I get a remote data engineering job from Africa?Yes, it is realistic, and data engineering is one of the skills global employers most often hire remotely from Africa. You need solid SQL and Python, one cloud platform, a pipeline tool and a warehouse, two or three end-to-end projects, and a search aimed at remote-friendly companies and talent platforms. Most remote hires have some experience, so a local first job helps.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com