Sefism.Data science: Towards ML engineering

Tauseef Fayyaz

Chapter 02

Statistics and probability you use

Summaries, distributions, uncertainty and tests: the part of statistics that decides whether a pattern in your data is real, and which test to reach for.

Mean, median, standard deviation and IQR

Also called descriptive statistics, variance, quartiles, spread

What: The mean is the sum over n; the median is the middle value; the standard deviation is the typical distance from the mean; the IQR is Q3 minus Q1, the spread of the middle half.
Why: They are the first thing to report about any column. The median and IQR stay honest when a few extreme values would drag the mean and standard deviation.
Example: Salaries 40, 45, 50, 55 and 400 (thousands): mean 118, median 50. The median describes the typical person; the mean describes the payroll.

Watch out

Reporting only the mean for skewed data such as income, order value or time on site.

Common distributions

Also called normal distribution, binomial, Poisson, exponential, Gaussian

What: Shapes that describe how a random quantity behaves: normal for sums of many small effects, binomial for counts of successes in n tries, Poisson for counts of events per interval.
Why: Choosing a test or a model means assuming a distribution, so you need to recognise which one your data resembles.
Example: Conversions out of 1,000 visitors are binomial; support tickets per hour are roughly Poisson; heights are roughly normal.

Watch out

Assuming everything is normal. Counts, durations and money are usually skewed and bounded at zero.

Conditional probability and Bayes' rule

Also called Bayes theorem, P(A|B), base rate, prior, posterior

What: P(A|B) is the probability of A given that B happened. Bayes' rule flips it: P(A|B) = P(B|A) P(A) / P(B).
Why: It explains why a 99% accurate fraud flag can still be wrong most of the time when fraud is rare: the base rate matters.
Example: Fraud rate 1%, flag catches 99% of fraud and wrongly flags 5% of normal orders: only about 1 in 6 flagged orders is fraud.

Watch out

Confusing P(A|B) with P(B|A), the base rate fallacy.

Sampling distribution and the CLT

Also called central limit theorem, CLT, law of large numbers, bootstrap

What: If you repeated your sample many times, the sample mean would vary; that spread is its sampling distribution. The central limit theorem says it is close to normal for large enough samples.
Why: It is why confidence intervals and t-tests work on data that is not itself normal, as long as the sample is not tiny or extremely skewed.

Watch out

Thinking the CLT makes the data normal. It only describes the average of repeated samples.

Standard error

Also called SE, standard error of the mean, SEM

What: The standard deviation of an estimate across repeated samples. For a mean, SE = s / sqrt(n); for a proportion, SE = sqrt(p(1 - p) / n).
Why: It turns a single number into a number with uncertainty. Quadrupling the sample only halves the standard error.

Watch out

Mixing up the standard deviation, which describes the data, with the standard error, which describes the estimate.

Confidence interval

Also called CI, 95% interval, margin of error

What: A range built so that 95% of intervals made this way would contain the true value. Roughly the estimate plus or minus 1.96 standard errors.
Why: It reports the size of an effect and its uncertainty together, which is more useful to a decision-maker than a p-value alone.

Python

from scipy import stats
res = stats.ttest_1samp(df["amount"], popmean=0)
ci = res.confidence_interval(confidence_level=0.95)
ci.low, ci.high

Watch out

Saying there is a 95% chance the true value is inside this particular interval; the 95% belongs to the method.

Hypothesis test and p-value

Also called null hypothesis, p value, significance, alpha

What: Assume no effect (the null hypothesis) and ask how surprising your data would be. The p-value is the probability of a result at least this extreme if the null were true.
Why: It is the standard gate for "is this difference likely noise", used in experiments, research papers and model comparisons.

Watch out

Reading p as the probability the null is true, or treating p = 0.049 and p = 0.051 as opposite conclusions.

Type I, type II errors and power

Also called false positive, false negative, statistical power, beta

What: A type I error is finding an effect that is not there (rate alpha, usually 5%). A type II error is missing a real one. Power is 1 minus the type II rate, usually targeted at 80%.
Why: Low-powered studies miss real effects and exaggerate the ones they find, so you plan sample size before collecting data.

Watch out

Concluding "no effect" from a small study that never had the power to detect one.

Which test when

Also called t-test, chi-square test, Mann-Whitney, ANOVA, paired t-test, statistical test table

What: Two group means: Welch t-test. Same people before and after: paired t-test. Two proportions: z-test or chi-square. Skewed or ranked data: Mann-Whitney U. Three or more means: ANOVA. Two numeric variables: correlation.
Why: Picking the test follows from the question and the data type, not from which one gives the smallest p-value.

Python

from scipy import stats
stats.ttest_ind(a, b, equal_var=False)  # Welch
stats.ttest_rel(before, after)          # paired
stats.mannwhitneyu(a, b)
stats.chi2_contingency(table)

Watch out

Running an independent t-test on paired data, which throws away the pairing and most of the power.

Correlation versus causation

Also called confounder, confounding, Pearson, Spearman, spurious correlation

What: Correlation measures how two variables move together, from -1 to 1. Causation needs more: an experiment, or a design that rules out confounders, a third factor driving both.
Why: Business decisions assume causes. Mistaking a correlation for one leads to changes that do nothing.
Example: Customers who use the app's wishlist spend more, but heavy shoppers are the ones who use wishlists: the habit drives both.

Watch out

Reporting Pearson correlation on curved or outlier-heavy data, where Spearman or a scatter plot would tell the truth.

Go deeper