Persistent Message Broker with Consumer Groups Built From Scratch

The problem: students use message queues without knowing what is inside

Almost every backend job description mentions Kafka, RabbitMQ or a cloud queue. Students learn to call a producer API and a consumer API, but when an interviewer or a production incident asks "what happens if the broker crashes after it received the message but before it wrote it to disk?" most cannot answer. The ideas that matter (append-only logs, offsets, fsync, consumer groups, rebalancing, at-least-once delivery) are rarely built end to end in a degree.

Small teams also face a practical version of this problem. Running a full Kafka cluster for a modest workload is heavy, and managed services cost money. A small, understandable broker is useful on its own.

What already exists and the gap

• Apache Kafka is the reference design: partitioned logs, consumer groups and replication, but large to run and hard to read as a learning codebase.
• Redpanda reimplements the Kafka protocol in C++ for performance, aimed at production operators.
• NATS JetStream adds persistence to NATS and is light to run, with its own model of streams and consumers.
• RabbitMQ is a queue-first broker with rich routing, a different model from a partitioned log.

This project does not try to beat any of them. The gap it fills is a from-scratch broker, small enough to read in a weekend, with measured behaviour under crashes. The value is in the evidence: a report that shows exactly what the durability settings cost and what they buy. This is the same learning pattern as the labs in MIT's distributed systems course and the storage projects in CMU's database course.

What you build

• Storage: each partition is a directory of segment files. Messages are appended with a length prefix and checksum. A sparse offset index maps offsets to file positions so fetches do not scan.
• Protocol: a small binary protocol over TCP with produce, fetch, commit and metadata requests. Write the spec first; clients follow it.
• Consumer groups: a coordinator tracks group members with heartbeats, assigns partitions, rebalances on join or failure, and stores committed offsets in an internal topic.
• Durability modes: acknowledge immediately, after fsync, or after a group fsync every N milliseconds.
• Operations: retention, metrics (messages in, bytes on disk, consumer lag) and a simple CLI.
• Demo app: an order-processing pipeline with two consumer groups (billing and email) reading the same topic independently.

The technically interesting part

The hard problem is the gap between "the broker said OK" and "the data is safe". You must decide when to acknowledge, handle a partially written record at the tail of a segment after a crash (checksums let you truncate it on restart), and make offset commits crash-safe so consumers neither skip nor endlessly replay.

Show this with a fault-injection harness: a script that produces numbered messages, kills the broker or a consumer at random moments, restarts it, and at the end checks that every acknowledged message was consumed at least once and counts duplicates. Run it hundreds of times per durability mode. Then pair it with a latency benchmark so the trade-off is visible: fsync on every message is safe and slow, batched fsync is fast with a small window of risk. Evaluators remember a graph that explains a design choice.

How to evaluate it

• Durability: zero lost acknowledged messages across at least 500 randomised crash runs in fsync modes; loss window measured in the fast mode.
• Delivery: duplicates per 10,000 messages after consumer crashes, before and after your commit strategy changes.
• Throughput and latency: messages per second and p99 latency across batch sizes 1, 10 and 100 and each durability mode.
• Rebalancing: time for a group to resume after a consumer dies, and lag during the pause.
• Reference comparison: the same benchmark against one existing broker on the same hardware, reported without spin.

Suggested plan

1. Weeks 1 to 3: read the Kafka design documentation and OSTEP chapters on file systems and crash consistency; write the protocol spec.
2. Weeks 4 to 8: segmented log, offset index, single-partition produce and fetch.
3. Weeks 9 to 12: topics, partitions, retention, metrics, first benchmark.
4. Semester 2, weeks 1 to 5: consumer groups, coordinator, committed offsets, rebalancing.
5. Weeks 6 to 10: fault-injection harness and durability modes.
6. Weeks 11 to 14: demo app, reference comparison, report and code walkthrough.

Path to a product

As a product this is an open source, single-binary broker for small teams and edge deployments that want durable queues without running Kafka. Monetise through support, a hosted version or consulting. Just as valuable, it is a teaching kit: universities and bootcamps can use your labs and fault tests, and it is a strong portfolio piece for backend and infrastructure roles.

Scope warning

Do not implement the full Kafka protocol, transactions or exactly-once semantics. Do not build multi-node replication before single-node durability is proven. Do not add a web dashboard beyond basic metrics; the storage engine is the project.

Learn what you need on Sefism

• System Design roadmap : queues, partitions and delivery guarantees in context.
• System Design cheat sheet : quick reference for consistency and messaging trade-offs.
• Apache Kafka Documentation : the design section explains the log, offsets and consumer groups you are rebuilding.
• Operating Systems: Three Easy Pieces : file systems and crash consistency chapters behind your fsync decisions.
• Confluent Developer Courses : see how consumer groups behave in practice before designing yours.

Get help with this idea

Broker projects impress evaluators when the crash tests are real and the benchmark is honest about what you did not build. If you want a second opinion on scope, stack or how to evaluate it before you pitch it to your supervisor, bring your draft proposal and we will pressure-test it together. You build it; the guidance is on the decisions.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session

New here? Create a free account first.