Trustworthy Online Controlled Experiments

What the book is

Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing is written by Ron Kohavi, Diane Tang and Ya Xu, who led experimentation work at Microsoft, Google and LinkedIn respectively. Cambridge University Press published it in March 2020. The authors draw on practical experience at companies that each run more than 20,000 controlled experiments a year, and share the examples, pitfalls and advice they collected.

What it covers

The book is organised in five parts that move from foundations to platforms and statistics. Topics include using the scientific method to evaluate ideas, defining key metrics and an overall evaluation criterion, testing whether results can be trusted, common pitfalls such as carryover effects and Twyman's law (any figure that looks interesting or different is usually wrong), the statistics behind the analysis, and building an experimentation platform that scales.

Is it worth it?

If you will ever analyse an A/B test, yes. Most courses teach the t-test and stop there. This book teaches what goes wrong in practice: sample ratio mismatch, metrics that move for the wrong reasons, peeking at results, novelty effects and interference between users. Those are the mistakes that turn a confident launch decision into a costly one, and they are rarely covered anywhere else in one place.

Where it fits for a data science learner

Read it after you are comfortable with basic statistics: sampling, confidence intervals and hypothesis tests. It is the natural next step when your roadmap reaches experiments, and it is essential reading for product analytics and data science roles at companies that ship software.

How long does it take?

At a steady pace, two to four weeks. The chapters are fairly self-contained, so you can read the introductory part first and return to specific chapters when a problem at work or in a project calls for them.

Prerequisites

Introductory statistics is enough to follow most of the book. The more statistical chapters assume you know what a p-value, variance and a confidence interval are. No programming is required.

Free vs paid

This is a paid book, sold by Cambridge University Press and other booksellers. Check your university library, which may provide access through Cambridge Core.

What you can do after it

Design an experiment with a clear hypothesis and sensible metrics, choose guardrail metrics, check that an experiment is trustworthy before reading its results, interpret the outcome without common statistical traps, and explain to a team why a surprising result deserves scepticism.

Common questions

Is it only for large tech companies? No. The principles apply to any product with users, though some platform chapters matter most at scale.

Is it very mathematical? It is practical first. The statistics are there, but the focus is on decisions and pitfalls.