A/B Test Analysis Report on a Public Experiment Dataset

The problem product teams have with experiments

Every product team eventually asks the same question: did the change work? A new checkout button, a different onboarding screen, a price shown in local currency. They run an A/B test, look at the dashboard, see that variant B is two percent higher, and ship it. Often the difference was noise, the groups were not split properly, or someone checked the result every morning and stopped the test on the first good day.

Answering the question well needs statistics, careful data checks and the discipline to write the decision rule before looking. Very few beginner portfolios show this, which is why it stands out, and why analysts at product companies spend so much of their time on it.

What already exists and the gap

• Optimizely and Statsig are commercial experimentation platforms with built-in statistics.
• GrowthBook is an open source feature flagging and experimentation platform that reads from your own warehouse.
• Kaggle hosts several public experiment datasets, such as a mobile game retention test and landing page conversion tests.
• Online calculators give a p-value for two conversion rates, and nothing about data quality.

Platforms automate the arithmetic, but they cannot tell you whether the question was set up well, whether users leaked between groups, or what to do with a result of "no detectable difference". Small teams without a platform often analyse tests in a spreadsheet with no checks at all. The gap is the judgement layer: a plan, checks and a memo a manager can act on.

What you build

1. A pre-analysis plan, committed to the repository with a date before any comparison: the primary metric, a guardrail metric (such as refunds or uninstalls), the test, the decision rule and the smallest effect that would matter to the business.
2. Data checks: duplicate users, users in both groups, missing timestamps, and a sample ratio mismatch test comparing group sizes with the planned split.
3. The estimate: difference in the primary metric with a 95 percent confidence interval, calculated in a spreadsheet and in code, with results that agree.
4. Robustness: the effect by day, to look for a novelty effect, and in one segment chosen in advance.
5. The memo: one page for a product manager: ship, do not ship or run longer, and why.

The technically interesting part

Two things make this more than arithmetic. The first is the sample ratio mismatch check: if a 50/50 split produced noticeably unequal groups, something in assignment or logging is broken, and the result cannot be trusted however significant it looks. Large experimentation teams treat this as a hard stop. The second is interpreting the interval, not the p-value. An interval of "minus 0.3 to plus 1.1 percentage points" tells a manager far more than "not significant": the change might help a little, might hurt slightly, and the test could not tell. Explaining that in plain words is the skill.

How to evaluate it

• Agreement: spreadsheet and code estimates match to the stated precision.
• Plan discipline: the plan's commit date precedes the analysis commits.
• Reader test: give the memo to 3 people who are not analysts and check they can state the decision and the main risk.
• Simulation (stretch): show the false positive rate at 5 percent with a fixed horizon and how much it rises with daily peeking over a few thousand simulated null tests.

Suggested plan

1. Week 1: choose the dataset, read its documentation and licence, write and commit the plan.
2. Week 2: data checks and the main estimate, two ways.
3. Week 3: robustness cuts, charts, memo draft.
4. Week 4: reader test, template clean-up, publish.

To grow this into a final year project, analyse three experiments from different contexts, build the reusable template, and add a simple web form that produces the checks and memo draft from an uploaded CSV.

Path to a product

The reusable template is the product. Small e-commerce stores and app studios run tests without an analyst, and a tool that runs the checks and writes a draft memo from a CSV is something they would use. Open source it first to earn trust, then offer a hosted version or a per-test analysis service. A realistic first customer is a local online store or a mobile game studio already running tests.

Scope warning

Do not build an experimentation platform with feature flags and user assignment; GrowthBook already exists. Do not use Bayesian and frequentist methods at once unless you can explain both. Note clearly if your dataset is simulated, as some public ones are.

Learn what you need on Sefism

• Data analytics roadmap : statistics, SQL and communication skills in order.
• Machine learning cheat sheet : a refresher on probability and evaluation terms.
• OpenIntro Statistics : a free textbook covering confidence intervals and tests for proportions.
• Kaggle Datasets : where to find public experiment data.
• Storytelling with Data : how to write the memo and charts for a busy reader.

Get help with this idea

A clear plan and honest checks matter more than fancy statistics here. Tauseef can help you choose a dataset, sharpen your decision rule and design a memo that shows real judgement. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session