Stage 1 · Lesson 7 of 8
Ingesting from APIs
Many data sources are , web addresses that return data when you ask. Payment providers, CRMs (customer relationship tools) and public data portals all work this way. Reading them well takes four skills.
First, : results come in pages, so you follow page numbers, cursors or next links. Second, rate limits: the API caps how often you can call it, and HTTP 429 means slow down, so you back off and wait. Third, authentication: keep API keys in or a .
Fourth, retries with exponential backoff, waiting longer after each failure, for short-lived errors. Save the raw response first, as JSON files grouped by date, before you parse anything. Then if your parsing logic changes, you can reprocess without calling the API again. Aim for : running it twice for the same day gives the same result.
Common traplooping until the API returns an empty page with no timeout, rate limit handling or logging, and silently loading half a day.
A question to check yourself, an AI practice session and 2 cheat sheet entries come with the full roadmap. Unlock
Resources for this stage
Start with· follow one of these
- CourseFreeData Engineering Zoomcamp
DataTalks.Club · Beginner
Free, cohort-based or self-paced course: Module 1 covers Docker, PostgreSQL and loading real taxi data
- CourseFreeCS50's Introduction to Programming with Python
Harvard University · Beginner
If your Python is shaky, work through the lectures on functions, exceptions, file I/O and libraries first