Stage 1 of 8 · 8 lessons · ~4 weeks
What data engineers do
Learn the data lifecycle, the main pipeline patterns and the Python you need to pull data from files, databases and APIs reliably, with a local setup to practise on.
Start: The data engineer's jobLessons· one lesson for each topic, in this order
Do this
- Write down the five lifecycle stages and one tool for each.
- Install Python, Git, VS Code, Docker and DuckDB, and run PostgreSQL in a container.
- Choose one public dataset or API you will reuse in every stage.
- Write an ingestion script that handles pagination, retries and rate limits, and stores raw files by date.
- Rerun the script for the same day and confirm the result is identical.
- Push the code with a README that explains how to run it.
Start with· follow one of these
- CourseFreeData Engineering Zoomcamp
DataTalks.Club · Beginner
Free, cohort-based or self-paced course: Module 1 covers Docker, PostgreSQL and loading real taxi data
- CourseFreeCS50's Introduction to Programming with Python
Harvard University · Beginner
If your Python is shaky, work through the lectures on functions, exceptions, file I/O and libraries first
Keep handy
Also worth your time
Before moving on: A Git repository with a Python script that pulls data from a public API or source database, saves the raw data as dated files and loads a cleaned table into DuckDB or PostgreSQL, with logging and a README.
Do not rush past this stage to Spark and Kafka. Most real data engineering work is careful ingestion and clean SQL, and the people who get hired can explain why a pipeline is safe to rerun. Pick one dataset now and keep using it: by the end you will have a portfolio project instead of ten half-finished tutorials.