Data resources for developers

A checklist for taking retrieval-augmented generation from demo to production. It covers ingestion and chunking, document permissions and injection, embeddings and indexes, hybrid search and reranking, grounded answers with citations, retrieval and answer evals, freshness, monitoring and cost, with a severity and a verification step for each check.

• AI
• Data

A checklist for taking a trained machine learning model into production and keeping it healthy there. It covers baselines and metrics, training and serving consistency, offline validation, packaging and the model registry, staged rollout and rollback, monitoring for skew and drift, and ownership, with a severity and a verification step for each check.

• Data
• DevOps

Best practices for building and evaluating a machine learning model before it goes anywhere near production, checked against the scikit-learn and PyTorch documentation, Google's Rules of Machine Learning and ML Crash Course, MLflow and Fairlearn. It covers problem framing and baselines, data splits and leakage, features, training and tuning, metrics and error analysis, reproducibility and experiment tracking, and fairness checks.

• Data
• Machine learning

Best practices for charts and data visualizations, checked against the UK Government Analysis Function's chart and colour guidance, WCAG 2.2, and the matplotlib, Plotly and Vega-Lite documentation. It covers choosing the chart, encoding and clutter, color and accessibility, axes and scales, titles and annotation, dashboards and interactive charts, and how to avoid charts that mislead.

• Data

A checklist for keeping warehouse and lakehouse tables trustworthy. It covers ownership and expectations, freshness, volume, schema, nulls, uniqueness and valid values, referential and reconciliation checks, anomaly monitoring and handling data incidents, with a severity and a way to verify each check.

• Data
• Testing

A practical checklist for engineers and analysts who store, query, move or share personal data. It covers knowing which rules apply, data inventory and mapping, minimization, PII classification, masking and pseudonymization, retention and deletion, access control, cross-border transfers, logging, vendor sharing and breach readiness. It is general guidance, not legal advice.

• Security
• Data

Best practices for batch data pipelines, checked against the Apache Airflow and dbt documentation and Google's guidance on data validation for ML systems. It covers idempotent, re-runnable tasks, orchestrator hygiene, layered transformations, data tests and freshness checks, ownership and lineage, and monitoring for silent failures.

• Data
• Databases

Best practices for modeling analytics data, checked against the Kimball Group's dimensional modeling techniques, Microsoft's dimensional modeling guidance and the dbt documentation. It covers the grain, fact and dimension tables, slowly changing dimensions, dbt marts and naming, tests and contracts, and defining metrics once in a semantic layer.

• Data
• Databases

Best practices for answering questions with data in SQL, pandas and Polars, checked against Google's Good Data Analysis guide and the pandas, Polars and PostgreSQL documentation. It covers framing the question, reading data correctly, joins and grain, aggregation pitfalls, reproducible notebooks, checking results and communicating findings.

• Data
• Databases

A checklist to run before you launch a business or product analytics dashboard. It covers purpose and audience, metric definitions, data freshness, correctness checks, performance, access control, accessibility, ownership and adoption, with a severity and a concrete way to verify each check.

• Data

The practical guide to A/B testing from experimentation leaders at Microsoft, Google and LinkedIn. It covers choosing metrics, designing experiments, spotting untrustworthy results and building an experimentation platform, drawn from companies that each run more than 20,000 experiments a year.

• Data

Posit's free, open source desktop IDE for R, with a console, editor, data viewer, plots pane, debugger, Git integration and a visual editor for Quarto and R Markdown in one window. It is the default place most people write R, and the one R for Data Science assumes you have open.

• Data

The book most people use to learn modern R: import, tidy, transform, visualise and communicate data with the tidyverse and Quarto. Written by Hadley Wickham, Mine Çetinkaya-Rundel and Garrett Grolemund, and free to read online, it takes you from zero to a reproducible analysis.

• Data
• Machine learning

Pakistan's national statistics office publishes census results, the Labour Force Survey, the Household Integrated Economic Survey, PSLM and monthly price indices, and releases anonymised survey microdata in SPSS and Stata formats free of charge. Real, messy, documented data for practising analysis on questions that matter.

• Databases
• Data

India's national open data portal, data.gov.in, brings together hundreds of thousands of datasets published by central ministries and state departments, mostly as CSV, with APIs for many of them. A huge source of real public data for practising cleaning, joining and analysis.

• Databases
• Data

An open standard for collecting data lineage: schedulers, warehouses and processing engines emit events about runs, jobs and datasets, so you can trace where data came from and what a change will affect.

• AI
• Data
• Databases

A free decision tree that helps you pick the right chart from the shape of your data, with explanations of each chart type, its common mistakes and code examples. Ideal when you are unsure whether a bar, line, scatter or something else fits.

• Data

A free, well-regarded introductory statistics textbook covering data, distributions, sampling, inference, hypothesis tests and regression, with videos, slides and labs. Everything an analyst needs to understand averages, uncertainty and A/B tests, explained without heavy maths.

• Data

A free, analysis-focused SQL tutorial that goes from SELECT to joins, subqueries and window functions, with a query editor in the browser. Originally Mode's well-known tutorial, it is now hosted by ThoughtSpot, which acquired Mode in 2023.

• Databases
• Data

Google's official list of every Google Sheets function, grouped by type with syntax and a link to a full help page for each. Keep it open while you work: it is the quickest way to check XLOOKUP, FILTER, UNIQUE, QUERY and the date and text functions.

• Data

Microsoft's official Excel help hub: short training, a full function reference, and step-by-step guides to XLOOKUP, PivotTables, data cleaning and charts. The most accurate free source for how Excel functions actually behave.

• Data

A free, non-profit site with thousands of interactive charts and downloadable datasets on health, population, energy, poverty, education and more. A great source of clean, well-documented public data for portfolio projects, and a model of clear, honest charts.

• Data

The best-known practical guide to turning charts into clear messages for business audiences: choosing the right visual, removing clutter, focusing attention and telling a story. The blog, podcast and monthly challenges are free; the 2015 book is worth buying if you present data often.

• Career
• Data

Tableau's official documentation, including a free step-by-step Get Started tutorial and the reference pages for calculated fields, table calculations and LOD expressions. The most reliable free way to learn how Tableau actually behaves.

• Data