DuckDB Documentation

What it is

DuckDB is an open source analytical (OLAP) database that runs inside another process, much like SQLite does for transactional work. You install it as a Python package or a single command-line binary, and it can query Parquet, CSV and JSON files directly, including files in S3, Google Cloud Storage or Azure storage through its extensions. It uses a columnar, vectorised engine, so aggregations over millions of rows are fast on a normal laptop. It is released under the MIT licence.

Why data engineers use it

• Learning: practise window functions, joins and aggregations on realistic data without a warehouse account.
• Local development: run dbt against DuckDB (the dbt-duckdb adapter) to build and test models for free before deploying to a cloud warehouse.
• Understanding formats: compare how long the same query takes on CSV and on Parquet, and see partition pruning with Hive-style partitioned folders.
• Small production jobs: many real datasets fit on one machine, and DuckDB can replace a Spark cluster for them.

The pages worth reading first

• Installation and the Python client (DuckDB works directly with pandas and Polars DataFrames).
• Data import: CSV, Parquet and JSON, reading many files with glob patterns, and partitioned writes.
• The SQL introduction and the pages on window functions and friendlier SQL extensions.
• Extensions: httpfs for S3 and HTTP, plus Iceberg and Delta for lakehouse tables.
• Performance guides, including how to profile a query.

Prerequisites

Basic SQL. Python helps but is not required; the CLI works on its own.

Is it worth your time?

Yes, early in your roadmap. It removes the setup cost of learning analytical SQL and file formats, and it is increasingly used in real pipelines and in courses such as the Data Engineering Zoomcamp. The trap is assuming it replaces a warehouse for everything: it is single-node, and concurrent writes from many users are not its job.

Common questions

DuckDB or SQLite? SQLite is built for transactional, row-by-row workloads; DuckDB is built for analytical queries over many rows and columns.

Is it free for commercial use? Yes, under the MIT licence.