Stage 1 · Lesson 3 of 8
Batch and streaming
processing handles data in chunks on a schedule. For example, every night you load yesterday's orders. Streaming handles each event within seconds of it happening.
For example, you flag a suspicious card payment right now. Batch is simpler to build, test, rerun and pay for, and it covers most reporting needs. Streaming is worth its extra complexity when a decision loses value within minutes.
Fraud checks, live stock levels and alerts are good examples. Many teams use a middle ground called micro-batch: small batch runs every few minutes. Before you choose, ask two questions.
How fresh does this really need to be? What happens if it is an hour late?
Common trapbuilding a streaming system because it sounds impressive, then maintaining it for a dashboard someone opens once a day.
A question to check yourself, an AI practice session and 1 cheat sheet entry come with the full roadmap. Unlock
Resources for this stage
Start with· follow one of these
- CourseFreeData Engineering Zoomcamp
DataTalks.Club · Beginner
Free, cohort-based or self-paced course: Module 1 covers Docker, PostgreSQL and loading real taxi data
- CourseFreeCS50's Introduction to Programming with Python
Harvard University · Beginner
If your Python is shaky, work through the lectures on functions, exceptions, file I/O and libraries first