Python for data

The everyday toolkit: NumPy arrays and vectorised maths, pandas for tables, notebooks for exploration, charts, and the file formats data arrives in.

NumPy arrays

Also called ndarray, NumPy

What: N-dimensional arrays holding one data type, with a shape and a dtype, stored in contiguous memory and operated on by fast compiled code.
Why: They are the common data format underneath pandas, scikit-learn and much of the scientific Python stack, and their speed makes numerical work in Python practical.

Python

import numpy as np

X = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
X.shape, X.dtype        # (3, 2), float64
X[:, 0]                 # first column
X[X[:, 1] > 3]          # rows where column 1 is above 3
X.mean(axis=0)          # one mean per column
X.reshape(2, 3)

Watch out

Mixing up axis=0 (down the rows, one result per column) and axis=1 (across the columns, one result per row).

Vectorisation and broadcasting

Also called broadcasting, vectorization

What: Vectorisation applies an operation to whole arrays at once instead of looping in Python. Broadcasting combines arrays of different shapes by stretching dimensions of size 1, such as subtracting a per-column mean from a matrix.
Why: Vectorised code is often 10 to 100 times faster and shorter than loops, and it is how ML maths is written in every library.

Python

rng = np.random.default_rng(0)
X = rng.normal(size=(1000, 3))
mu, sigma = X.mean(axis=0), X.std(axis=0)   # shape (3,)
X_std = (X - mu) / sigma                     # shape (1000, 3)
sq_dists = ((X[:, None, :] - X[None, :, :]) ** 2).sum(-1)

Watch out

Accidentally broadcasting shapes (n,) and (n, 1) into an (n, n) matrix: check shapes whenever a result looks odd.

pandas DataFrames

Also called pandas, DataFrame, Series

What: A table of labelled columns, each a typed Series, with a row index. It supports loading, filtering, joining, grouping and reshaping data.
Why: Most tabular ML work, from loading raw data to building features, happens in pandas before the data is handed to a model.

Python

import pandas as pd

df = pd.read_csv("orders.csv", parse_dates=["created_at"])
df.info()
df.describe()
mask = df["created_at"] >= "2026-01-01"
recent = df.loc[mask, ["user_id", "amount"]]
df.loc[df["amount"] < 0, "amount"] = np.nan

Watch out

Chained assignment such as df[mask]["amount"] = 0, which does not update df: use df.loc[mask, "amount"] = 0.

groupby, merge and reshaping

Also called group by, join, pivot table, aggregation

What: groupby splits rows into groups and aggregates each one; merge joins tables on keys like SQL; pivot_table and melt switch between wide and long formats.
Why: Features such as "orders per user in the last 30 days" are built from exactly these operations.

Python

per_user = (
    df.groupby("user_id")
    .agg(orders=("order_id", "count"), spend=("amount", "sum"))
    .reset_index()
)
users = users.merge(
    per_user, on="user_id", how="left", validate="one_to_one"
)
cols = ["orders", "spend"]
users[cols] = users[cols].fillna(0)

Watch out

Merging on keys that are not unique and silently multiplying rows: pass validate to catch it.

Jupyter notebooks

Also called notebook, ipynb, JupyterLab, Colab

What: Interactive documents that mix code cells, output, charts and notes, run by a kernel that keeps variables alive between cells.
Why: They are ideal for exploring data and sharing analysis. Move stable code into modules and scripts once it becomes part of a training pipeline.

Watch out

Running cells out of order so the notebook only works in your session: restart and run all before sharing.

Matplotlib charts

Also called Matplotlib, plotting, seaborn, data visualisation

What: The base plotting library for Python; seaborn adds statistical charts on top. The object-oriented style creates a figure with axes and draws on each axis.
Why: Plots reveal distributions, outliers, relationships and training curves that summary numbers hide.

Python

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].hist(df["amount"], bins=50)
axes[0].set(title="Order amount", xlabel="GBP")
axes[1].scatter(df["age"], df["amount"], s=5, alpha=0.3)
axes[1].set(xlabel="Age", ylabel="Amount")
fig.tight_layout()

CSV, JSON and Parquet

Also called Parquet, file formats, JSON Lines

What: CSV is plain-text rows with no types; JSON holds nested records, often one per line (JSON Lines); Parquet is a compressed, columnar binary format that stores the schema.
Why: Parquet loads far faster, is much smaller and keeps types such as dates and categories, so it suits datasets beyond a few megabytes. CSV and JSON suit exchange with other systems.

Python

df = pd.read_csv("raw.csv", dtype={"zip_code": "string"})
events = pd.read_json("events.jsonl", lines=True)
df.to_parquet("clean.parquet", index=False)
cols = ["user_id", "amount"]
df = pd.read_parquet("clean.parquet", columns=cols)

Watch out

Letting CSV readers parse ids such as zip codes or phone numbers as numbers, which drops leading zeros.

Plain Python for data work

Also called lists, dicts, comprehensions, functions

What: Before NumPy and pandas come lists, dicts, comprehensions, functions, f-strings and reading files with with open(...).
Why: Data loading, feature logic and glue code are plain Python, and knowing when to swap a Python loop for a vectorised call starts here.

Python

rows = [{"city": "Lahore", "temp": 34}, {"city": "Oslo", "temp": 12}]
hot = [r["city"] for r in rows if r["temp"] > 30]
by_city = {r["city"]: r["temp"] for r in rows}

def mean(xs):
    return sum(xs) / len(xs) if xs else 0.0

with open("data.csv") as f:
    header = f.readline().strip().split(",")
print(f"{len(rows)} rows, mean {mean(list(by_city.values())):.1f}")

Watch out

Looping over millions of rows in plain Python when one NumPy or pandas operation does it far faster.

Go deeper