Sefism.Data engineering: AWS

Tauseef Fayyaz

Chapter 01

Foundations and ingestion

What data engineers do, the lifecycle every platform follows, the main pipeline patterns and how to pull data from sources safely.

Data engineer

Also called DE, analytics engineer, data platform engineer

What: An engineer who builds and runs the systems that collect data from sources, store it, transform it into reliable tables and serve it to analysts, data scientists and products.
Why: Every dashboard, model and data product depends on this work. The role mixes software engineering habits with an understanding of how the business uses data.

Watch out

Collecting tools instead of mastering SQL, Python, modelling and reliable pipeline design, which is what hiring teams test.

Data engineering lifecycle

Also called data lifecycle, source systems

What: The stages data passes through: generation in source systems, ingestion, storage, transformation and serving, with security, quality, orchestration and cost underneath all of them.
Why: It is a map for designing any platform: start from what the consumer needs and how fresh it must be, then work back to the sources.
Example: App database and payment API (sources), nightly load (ingestion), Parquet in object storage (storage), dbt models (transformation), a BI dashboard (serving).

Watch out

Choosing tools first and discovering later that nobody needed the data that fresh or in that shape.

Batch and streaming

Also called micro-batch, real time

What: Batch processes data in chunks on a schedule, such as yesterday's orders every night. Streaming processes each event within seconds of it happening.
Why: Batch is simpler, cheaper and easier to rerun, and covers most reporting. Streaming earns its complexity only when a decision loses value within minutes.

Watch out

Building a streaming system for a report someone reads once a day.

ETL and ELT

Also called extract transform load, extract load transform, ETL, ELT

What: ETL transforms data before loading it into the warehouse. ELT loads raw data first and transforms it inside the warehouse with SQL.
Why: ELT is the modern default because cloud storage is cheap and keeping raw data lets you rebuild models when logic changes. ETL still fits when data must be masked before it lands.

Watch out

Loading raw data and never modelling it, so every analyst cleans the same mess differently.

Full and incremental loads

Also called full load, incremental load, delta load, high watermark

What: A full load copies the whole source table each run. An incremental load copies only rows changed since the last run, usually filtered on an updated_at column or increasing ID.
Why: Full loads are simple and correct for small tables; incremental loads keep large tables fast and cheap to sync.

Watch out

Relying on an updated_at column the application does not always set, or missing hard deletes entirely.

Change data capture

Also called CDC, Debezium, log-based replication

What: Reading a database's own change log, such as the PostgreSQL write-ahead log or MySQL binlog, to capture every insert, update and delete in order.
Why: It sees deletes and every intermediate change with little load on the source, and can feed both a warehouse and a stream.

Watch out

Letting the replication slot or log fall behind unmonitored, which can fill the source database's disk.

API ingestion

Also called REST ingestion, pagination, rate limiting, backoff

What: Pulling data from an HTTP API while handling pagination, rate limits (HTTP 429), authentication and retries with exponential backoff.
Why: Many sources are SaaS APIs. Saving the raw response first lets you reprocess without calling the API again.

Python

import requests
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(5), wait=wait_exponential(max=60))
def get_page(url, params):
    r = requests.get(url, params=params, timeout=30)
    r.raise_for_status()  # raises on 429 and 5xx, so it is retried
    return r.json()

Watch out

Looping until an empty page with no timeout, retries or logging, and silently loading half the data.

Python for pipelines

Also called pipeline script, logging, virtual environment

What: Python written to run unattended: small functions, configuration from environment variables, logging, clear exceptions, pinned dependencies and a non-zero exit code on failure.
Why: Pipelines run at night with nobody watching. Logs and exit codes are how the orchestrator and you find out what went wrong.

Watch out

Scheduling a notebook with print statements and hard-coded credentials.

DuckDB

Also called in-process OLAP, pandas, Polars

What: A small, in-process analytical database that runs SQL directly over CSV, JSON and Parquet files on your laptop, much faster than pandas on larger data.
Why: It behaves like a cloud warehouse at zero cost, so you can practise modelling and pipelines locally before paying for the cloud.

SQL

SELECT country, sum(amount) AS revenue
FROM 'data/silver/orders/*/*.parquet'
WHERE order_date >= DATE '2026-09-01'
GROUP BY country
ORDER BY revenue DESC;

Watch out

Reaching for Spark for a few gigabytes that DuckDB handles on one machine in seconds.

Idempotent pipeline

Also called idempotence, safe rerun, effectively once, idempotent, idempotency

What: A job that produces the same result whether it runs once or many times for the same input, for example by overwriting a date partition or using MERGE instead of INSERT.
Why: Retries, backfills and duplicate events all mean jobs run more than once. Idempotency makes that harmless.
Example: A daily job deletes and rewrites the 2026-09-01 partition, so a retry after a crash leaves exactly one day of rows, not two.

Watch out

Appending on every run, so each retry quietly duplicates rows.

Go deeper