Data science: Business data science

Sefism.

Chapter 01

Code for data: Python or R

The working set you touch every day: notebooks, arrays, data frames, filtering, grouping, joining and reshaping, plus an environment someone else can rebuild.

Notebook workflow

Also called Jupyter, JupyterLab, RStudio, Quarto, Colab

What
Code, output, charts and notes in one document you run cell by cell: Jupyter or Colab for Python, RStudio with Quarto or R Markdown for R.
Why
Exploration is a loop of try, look, adjust. A notebook keeps the evidence next to the code, and a clean one becomes the report.

Watch out

Running cells out of order until the notebook only works in your session. Restart and run all before you trust or share it.

Arrays and vectors

Also called NumPy array, ndarray, R vector, dtype

What
A block of values of one type: a NumPy array in Python, an atomic vector in R. Maths applies to the whole block at once.
Why
Every data frame column is one of these underneath, so their rules on types, missing values and broadcasting explain most surprises.
Python
import numpy as np
x = np.array([3, 5, 8])
x * 2          # array([ 6, 10, 16])
x.mean(), x.std(ddof=1)

Watch out

Forgetting that NumPy std divides by n by default while R sd and pandas divide by n minus 1.

Vectorised operations

Also called vectorization, broadcasting

What
Writing one expression over a whole column instead of looping over rows, so the work runs in fast compiled code.
Why
It is often a hundred times faster than a Python loop, and shorter to read. Loops over rows are the usual sign of beginner data code.
Example
Price with tax for a million rows: df["price"] * 1.18 in one line, instead of a for loop that appends to a list.

Watch out

Reaching for apply with a Python function, which is still a hidden loop, when a built-in column operation exists.

DataFrame and tibble

Also called pandas DataFrame, tibble, data.frame, read_csv

What
A table in memory: named columns, each with one type, and rows that line up across them. pandas in Python, tibbles in R.
Why
Almost every analysis starts by loading a file into one and checking its shape, types and first rows.
Python
import pandas as pd
df = pd.read_csv("sales.csv", parse_dates=["date"])
df.shape
df.info()      # columns, types, non-null counts
df.head()

Watch out

Skipping the type check and discovering later that a numeric column was read as text because of one stray value.

Selecting and filtering rows

Also called loc, query, filter, select, boolean indexing

What
Keeping the columns you need and the rows that meet a condition: loc and boolean masks in pandas, select and filter in dplyr.
Why
Most questions start by narrowing the data to the population you care about, such as last quarter's paying customers.
Python
paid = df.loc[
    (df["status"] == "paid") & (df["amount"] > 0),
    ["order_id", "region", "amount"],
]

Watch out

In pandas, using and instead of &, or leaving out the brackets around each condition.

Group by and aggregate

Also called groupby, group_by, summarise, split-apply-combine

What
Splitting rows into groups by one or more columns, computing a summary for each group and combining the results into a new table.
Why
Counts, averages and totals per segment answer most business questions and feed most charts.
Python
summary = (
    df.groupby("region")
      .agg(orders=("order_id", "count"),
           avg_amount=("amount", "mean"))
      .reset_index()
)

Watch out

Averaging averages: the mean of per-region means is not the overall mean when regions have different sizes.

Merging tables

Also called merge, join, left_join, left join

What
Combining two data frames on a key column, like a SQL join: merge in pandas, left_join and friends in dplyr.
Why
Real data arrives in pieces, such as orders, customers and products, and you join them to answer anything interesting.
Python
out = orders.merge(customers, on="customer_id",
                   how="left", validate="many_to_one")
len(out) == len(orders)   # should be True

Watch out

Joining on a key that is not unique on the lookup side, which silently duplicates rows and inflates every total.

Long and wide format

Also called melt, pivot, pivot_longer, pivot_wider, reshape

What
Wide data has one column per period or category; long data has one row per observation with a column saying which one. You convert between them.
Why
Plotting libraries and models mostly want long data, while people read wide tables. Reshaping is a daily step between the two.
Python
long = wide.melt(id_vars="store",
                 var_name="month", value_name="sales")
back = long.pivot(index="store", columns="month",
                  values="sales")

Watch out

Pivoting wide when two rows share the same index and column, which errors or quietly aggregates.

Reproducible environment

Also called virtual environment, venv, conda, renv, renv.lock, requirements.txt

What
A per-project record of the exact package versions an analysis used, so it runs the same on another machine or next year.
Why
Results that cannot be rerun cannot be trusted or reviewed. Hiring managers notice a repository that installs in one command.
Python
python -m venv .venv
source .venv/bin/activate
pip install pandas scikit-learn statsmodels
pip freeze > requirements.txt

Watch out

Installing everything into one global environment until two projects need different versions of the same package.

Go deeper

01 · Code for data: Python or R1 / 11