Stage 2 · Lesson 9 of 9
Hypothesis tests, p-values and confidence intervals
A asks whether a difference you saw could just be noise. Say a new model lifts clicks from 4.0% to 4.3%. You start by assuming there is no real difference; this is the .
Then you compute the p-value: how likely a result at least this extreme would be if the null were true. A confidence interval gives a range of believable values for the true effect, such as a lift of 0.1 to 0.5 points. This is how you judge A/B tests and model comparisons.
Common trapThe traps matter more than the formulas. A p-value is not the chance that the null is true. A result with p < 0.05 is not necessarily large or useful. And if you test twenty metrics, one will look significant by chance.
A question to check yourself, an AI practice session and 1 cheat sheet entry come with the full roadmap. Unlock
Resources for this stage
Start with· follow one of these
- CoursePaidMathematics for Machine Learning and Data Science Specialization
DeepLearning.AI · Beginner · Coursera subscription; financial aid available
Paid, with financial aid: linear algebra, calculus and statistics taught visually with Python labs
- CourseFreeKhan Academy Statistics and Probability
Khan Academy · Beginner
Free, complete statistics and probability with practice questions on every topic
Keep handy
Also worth your time
- Mathematics for Machine LearningGo deeper
Free book that ties the maths directly to regression, PCA and classifiers