Stage 1 · Lesson 5 of 8
Ingestion: full loads, incremental loads and CDC
There are three common ways to copy data from a source. A copies the whole table every run. It is simple and correct, but it gets slow and expensive as the table grows.
An incremental load copies only the rows that changed since the last run. It usually filters on an updated_at column or an ID that keeps increasing. This is faster, but it misses rows that were deleted, and it relies on that column always being set.
(CDC) reads the database's own change log, a record the database keeps of every edit. So it sees every insert, update and delete, in order, with little extra load on the source. Tools such as Debezium, or connectors your cloud manages for you, do CDC. Pick the simplest method that keeps your data fresh and correct enough.
Common trapan incremental load on a column that the application does not always update.
A question to check yourself, an AI practice session and 2 cheat sheet entries come with the full roadmap. Unlock
Resources for this stage
Start with· follow one of these
- CourseFreeData Engineering Zoomcamp
DataTalks.Club · Beginner
Free, cohort-based or self-paced course: Module 1 covers Docker, PostgreSQL and loading real taxi data
- CourseFreeCS50's Introduction to Programming with Python
Harvard University · Beginner
If your Python is shaky, work through the lectures on functions, exceptions, file I/O and libraries first