Apache Kafka Documentation

What it is

Apache Kafka is an open source distributed event streaming platform. Producers write events to topics, topics are split into partitions that are replicated across brokers, and consumers read events in order within each partition, tracking their position with offsets. Kafka Connect moves data in and out of other systems, and Kafka Streams processes streams inside Java applications.

Version

As of 2026-09 the documentation covers Kafka 4.3. Kafka 4.0, released in March 2025, was the first major release to run entirely without Apache ZooKeeper: clusters now use KRaft mode for metadata. Older tutorials that start a ZooKeeper process or use the --zookeeper flag are out of date; use --bootstrap-server instead.

The sections worth reading

• Introduction and Use Cases: what event streaming is for, in plain language.
• Quickstart: start a local broker (or use the official Docker image), create a topic, produce and consume events.
• Design: persistence, the log, replication, delivery semantics and why ordering is only per partition. This section explains at-least-once and exactly-once delivery better than most blog posts.
• Configuration and Operations, later, when you run Kafka for real.
• Kafka Connect, for ingesting from databases (change data capture) and writing to storage.

Prerequisites

Comfortable with the command line and one programming language. For data engineering, Python clients are common, while Kafka Streams runs on the JVM (Java or Scala). Understand batch pipelines first: streaming adds time, ordering and duplicates on top of everything a batch pipeline already has to get right.

Is it worth your time?

Yes, at concept level for everyone and in depth if you target streaming roles. Many cloud services (Amazon MSK, Azure Event Hubs with its Kafka-compatible endpoint, Confluent Cloud) speak the Kafka protocol, so the concepts transfer. Many junior data engineering roles are mostly batch, so do not spend months on Kafka before your SQL, modelling and orchestration are solid.

How to use it well

Run a single-broker cluster locally, write a small producer and consumer in Python, then break things on purpose: stop a consumer, restart it and watch it resume from its committed offset; add partitions and see how keys map to them; replay a topic from the beginning.

Common questions

Is Kafka free? Yes, it is open source under Apache 2.0. Managed services charge for hosting.

Kafka or a cloud-native stream service? Learn Kafka concepts first; Amazon Kinesis, Google Pub/Sub and Azure Event Hubs use the same ideas under different names.