Sefism.Data science: Statistics and R

Tauseef Fayyaz

Chapter 01

Code for data: Python or R

The working set you touch every day: notebooks, arrays, data frames, filtering, grouping, joining and reshaping, plus an environment someone else can rebuild.

Notebook workflow

Also called Jupyter, JupyterLab, RStudio, Quarto, Colab

What: Code, output, charts and notes in one document you run cell by cell: Jupyter or Colab for Python, RStudio with Quarto or R Markdown for R.
Why: Exploration is a loop of try, look, adjust. A notebook keeps the evidence next to the code, and a clean one becomes the report.

Watch out

Running cells out of order until the notebook only works in your session. Restart and run all before you trust or share it.

Arrays and vectors

Also called NumPy array, ndarray, R vector, dtype

What: A block of values of one type: a NumPy array in Python, an atomic vector in R. Maths applies to the whole block at once.
Why: Every data frame column is one of these underneath, so their rules on types, missing values and broadcasting explain most surprises.

x <- c(3, 5, 8)
x * 2          # 6 10 16
mean(x); sd(x)

Watch out

Forgetting that NumPy std divides by n by default while R sd and pandas divide by n minus 1.

Vectorised operations

Also called vectorization, broadcasting

What: Writing one expression over a whole column instead of looping over rows, so the work runs in fast compiled code.
Why: It is often a hundred times faster than a Python loop, and shorter to read. Loops over rows are the usual sign of beginner data code.
Example: Price with tax for a million rows: df["price"] * 1.18 in one line, instead of a for loop that appends to a list.

Watch out

Reaching for apply with a Python function, which is still a hidden loop, when a built-in column operation exists.

DataFrame and tibble

Also called pandas DataFrame, tibble, data.frame, read_csv

What: A table in memory: named columns, each with one type, and rows that line up across them. pandas in Python, tibbles in R.
Why: Almost every analysis starts by loading a file into one and checking its shape, types and first rows.

library(tidyverse)
df <- read_csv("sales.csv")
dim(df)
glimpse(df)    # columns and types
head(df)

Watch out

Skipping the type check and discovering later that a numeric column was read as text because of one stray value.

Selecting and filtering rows

Also called loc, query, filter, select, boolean indexing

What: Keeping the columns you need and the rows that meet a condition: loc and boolean masks in pandas, select and filter in dplyr.
Why: Most questions start by narrowing the data to the population you care about, such as last quarter's paying customers.

paid <- df |>
  filter(status == "paid", amount > 0) |>
  select(order_id, region, amount)

Watch out

In pandas, using and instead of &, or leaving out the brackets around each condition.

Group by and aggregate

Also called groupby, group_by, summarise, split-apply-combine

What: Splitting rows into groups by one or more columns, computing a summary for each group and combining the results into a new table.
Why: Counts, averages and totals per segment answer most business questions and feed most charts.

summary <- df |>
  group_by(region) |>
  summarise(orders = n(),
            avg_amount = mean(amount, na.rm = TRUE))

Watch out

Averaging averages: the mean of per-region means is not the overall mean when regions have different sizes.

Merging tables

Also called merge, join, left_join, left join

What: Combining two data frames on a key column, like a SQL join: merge in pandas, left_join and friends in dplyr.
Why: Real data arrives in pieces, such as orders, customers and products, and you join them to answer anything interesting.

out <- orders |>
  left_join(customers, by = join_by(customer_id),
            relationship = "many-to-one")
nrow(out) == nrow(orders)

Watch out

Joining on a key that is not unique on the lookup side, which silently duplicates rows and inflates every total.

Long and wide format

Also called melt, pivot, pivot_longer, pivot_wider, reshape

What: Wide data has one column per period or category; long data has one row per observation with a column saying which one. You convert between them.
Why: Plotting libraries and models mostly want long data, while people read wide tables. Reshaping is a daily step between the two.

long <- wide |>
  pivot_longer(-store, names_to = "month",
               values_to = "sales")
back <- long |>
  pivot_wider(names_from = month, values_from = sales)

Watch out

Pivoting wide when two rows share the same index and column, which errors or quietly aggregates.

Reproducible environment

Also called virtual environment, venv, conda, renv, renv.lock, requirements.txt

What: A per-project record of the exact package versions an analysis used, so it runs the same on another machine or next year.
Why: Results that cannot be rerun cannot be trusted or reviewed. Hiring managers notice a repository that installs in one command.

install.packages("renv")
renv::init()       # project library
renv::snapshot()   # write renv.lock
renv::restore()    # rebuild elsewhere

Watch out

Installing everything into one global environment until two projects need different versions of the same package.

The pipe and dplyr verbs

Also called |>, %>%, native pipe, dplyr, mutate, arrange

What: The pipe |> passes the result on the left as the first argument on the right, so an analysis reads as a chain of verbs: filter, mutate, group_by, summarise, arrange.
Why: Each step is one small, named transformation, which makes long analyses readable and easy to check line by line.

df |>
  filter(!is.na(score)) |>
  mutate(passed = score >= 50) |>
  group_by(cohort) |>
  summarise(pass_rate = mean(passed)) |>
  arrange(desc(pass_rate))

Watch out

Chains of thirty steps with no intermediate checks. Break them and look at the data between stages.

Go deeper