Real-Time Event Pipeline with Kafka and Windowed Metrics

The problem with streaming demos

Most streaming tutorials assume every event arrives once, in order and on time. Real streams do not behave like that. A delivery rider's phone loses signal in a basement and uploads twenty minutes of location pings at once. A payment service retries a request and the same "order paid" event is published twice. A consumer crashes halfway through a batch and restarts from its last committed offset. A dashboard built on the happy path shows wrong numbers on exactly the busy days when people are watching it.

Food delivery, ride-hailing and e-commerce companies all run pipelines like this for live orders per minute, rider availability or payment failure rates. Handling duplicates, late data and restarts correctly is what separates a working pipeline from a demo, and it is what data engineering interviewers probe.

What already exists and the gap

• Apache Kafka with Kafka Streams gives durable, partitioned logs and a library for stateful processing inside Java applications.
• Apache Flink offers event-time windows, watermarks and exactly-once state snapshots, and is widely used for this kind of workload.
• Spark Structured Streaming provides micro-batch streaming with watermarks, familiar to anyone who knows Spark.
• Managed services such as Amazon Kinesis, Google Pub/Sub and Confluent Cloud remove the operations work but bill while idle.

The tools are mature. What is missing, in most student portfolios and many junior engineers' heads, is evidence: a pipeline that has been deliberately broken and measured. This project is a correctness lab with a realistic domain, not a new streaming engine.

What you build

Pick one domain and keep it small, for example a food delivery app with three event types: order placed, rider assigned, order delivered.

• Simulator: generates keyed events with unique IDs and event timestamps, and has switches for duplicates, bursts and late events.
• Kafka: runs in Docker, topics partitioned by a sensible key (order ID or city) and a schema per event type.
• Stream job: deduplicates on event ID and computes windowed metrics by event time, such as orders per city per minute and average delivery time over a sliding 15-minute window, with a watermark.
• Sink and dashboard: results written idempotently (upsert on window and key) to PostgreSQL or ClickHouse, and shown in Grafana.
• Archive and reconciliation: every raw event also lands in object storage, and a batch job recomputes the metrics so you can compare.

The technically interesting part

The hard problem is correctness under failure. Your supervisor should see you explain, with numbers, the difference between processing time and event time, what a watermark trades off (lower latency against dropping more late events), and why an idempotent sink turns at-least-once delivery into correct results. Kill the job mid-window, restart it and show the dashboard does not double count. Push events past the watermark and show exactly where they go: dropped, sent to a side output, or corrected by the batch job. The reconciliation between streaming and batch is your proof.

How to evaluate it

• Duplicates: inject 5 percent duplicates and show the metric error before and after deduplication.
• Restarts: kill the job 20 times at random points and report any difference against the batch recomputation.
• Late data: vary the watermark (30 seconds, 2 minutes, 10 minutes) and plot dropped events against output delay.
• Throughput and lag: events per second sustained on a laptop and consumer lag under a burst.

Suggested plan

1. Weeks 1 to 2: domain, event schemas, simulator, Kafka in Docker.
2. Weeks 3 to 5: first stream job with windows and a sink, dashboard.
3. Weeks 6 to 8: deduplication, watermarks, idempotent writes, archive.
4. Weeks 9 to 11: batch reconciliation and the failure experiments.
5. Weeks 12 to 15: load tests, report and demo.

Path to a product

This is mainly a portfolio centrepiece for data and platform engineering roles, where it gives you a concrete story for system design rounds. It can also become a small open source template ("a tested streaming starter with reconciliation") that other teams clone, or the basis of consulting work for a local e-commerce or logistics company that wants live operations metrics without a full data team. The first real user could be a startup whose dashboard currently polls the production database.

Scope warning

Do not build a custom streaming engine, a multi-region cluster or a machine learning model on the stream. Run locally in Docker by default; if you use a cloud service, set a budget alert and delete capacity after each session. Keep two or three event types and put your effort into correctness, not features.

Learn what you need on Sefism

• System design roadmap : partitions, delivery semantics and idempotency in a wider context.
• System design cheat sheet : quick reference for queues, logs and consistency trade-offs.
• Apache Kafka Documentation : topics, partitions, consumer groups and offsets.
• Confluent Developer Courses : structured lessons on Kafka and stream processing.
• Fundamentals of Data Engineering : where streaming fits in the data lifecycle.

Get help with this idea

The risk here is spending the semester on setup and never running the failure experiments. Tauseef can help you cut the scope to what you can prove, pick between Flink and Spark for your team, and design an evaluation that examiners will respect. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session