Notification Service with Retries, a Dead Letter Queue and Idempotency Keys
Almost every product sends emails, SMS and push messages, and almost every team learns the hard way that providers time out, workers crash mid-job and a retry can send the same message twice. Build a standalone notification service with a queue, retries with exponential backoff, a dead letter queue, idempotency keys and a dashboard that shows the state of every delivery. The project is about failure handling you can demonstrate, not about sending a message.
Intermediate
A small team, or one strong student willing to learn something new.1 semester · 2 to 3 students
7 to ship
4 optional extrasSuggested stack
What you should ship
- An HTTP API that accepts notification requests with idempotency keys and returns the original result for a repeated key
- Queue-backed workers for at least two channels that can be killed at any point without losing messages
- Retries with exponential backoff and jitter, retryable and permanent error classes, and a dead letter queue with replay
- A fake provider with configurable latency, error rate and outage mode
- Per-user channel opt-outs and quiet hours respected by the workers
- A delivery dashboard showing each notification's state and attempt history
- A failure test plan covering provider outage, killed workers, duplicate requests and poison messages, demonstrated live
If you have time left
- Integration with a real email or SMS provider behind the same interface
- Per-tenant rate limits so one noisy application cannot delay the others
- Scheduled notifications and digest batching
- Webhook callbacks to the calling application when a delivery completes or fails
Problem
Sending one email is easy. Sending thousands reliably is not. Providers return errors, rate limit you, and sometimes accept a message and then time out before replying. A naive service either loses messages or, after a retry, sends a fee reminder to the same student three times. Teams usually discover this in production.
What you build
A service that other applications call over HTTP to send a notification:
- An API that accepts a request with a channel (email, SMS or web push), a template, a recipient and an idempotency key.
- A queue and a pool of workers, one worker type per channel.
- Retries with exponential backoff and jitter, a maximum attempt count, and a dead letter queue for messages that keep failing.
- User preferences: opt-outs per channel and quiet hours.
- A delivery dashboard listing every notification with its state (queued, sending, delivered, failed or dead), its attempt history, and a way to replay dead messages.
Suggested architecture
- API: Node.js with TypeScript. Store the request in PostgreSQL first, then enqueue it. If the same idempotency key arrives twice, return the original result instead of creating a second notification.
- Queue: RabbitMQ, or Redis Streams if you want fewer moving parts. Pick one and justify it.
- Workers: stateless and safe to kill at any moment. Acknowledge a message only after the send is recorded.
- Providers: use Mailpit or a similar local SMTP catcher for email, and write a fake SMS provider with switches that make it slow, flaky or down. Real providers are a stretch goal.
- Delivery guarantee: aim for at-least-once delivery with deduplication, and explain in the report why exactly-once delivery cannot be promised across a network.
Milestones by month
- Month 1: API, data model, idempotency keys, a single email worker without a queue.
- Month 2: queue, multiple workers, retries with backoff, dead letter queue.
- Month 3: fake provider with failure modes, preferences and quiet hours, delivery dashboard.
- Month 4: failure test plan, load test, report and demo.
How it will be assessed
A written failure test plan, executed live in the demo:
- the provider returns errors for five minutes, then recovers;
- a worker is killed while holding a message;
- the same request is submitted twice with the same idempotency key;
- a malformed message that can never succeed (a poison message) arrives.
For each case, show that nothing was lost, nothing reached the end user more often than your stated guarantee allows, and the dashboard told the truth. Include throughput and queue depth graphs from a load test.
Pitfalls
- Retrying immediately and forever, which turns a provider outage into a retry storm.
- Retrying errors that will never succeed, such as an invalid email address. Classify errors as retryable or permanent.
- Storing idempotency keys without an expiry or without the original response.
- Spending a month on HTML email templates. Plain templates are fine.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.