Sefism is free for learners. We are here to help you grow, one step at a time.Join for free
Sefism.
Other

Distributed Key-Value Store with Raft Consensus

An advanced distributed systems final year project: implement Raft from the paper and build a replicated key-value store on top that stays consistent through node crashes and network partitions, proven by a fault injection harness and a linearizability checker. A portfolio centrepiece for infrastructure and backend roles, and a teaching tool for a distributed systems course.

Topics
Difficulty

Advanced

Needs a team, real planning, and a supervisor who knows the area.
Effort

1 semester · 2 to 3 students

Deliverables

8 to ship

3 optional extras

Suggested stack

GogRPC and Protocol Buffersbbolt or a write-ahead log fileDocker ComposePorcupine linearizability checkerPrometheus metricsGitHub Actions
A suggestion, not a requirement. Swap anything for what you already know.

What you should ship

  • Raft leader election, log replication and commit rules implemented exactly as in Figure 2 of the extended paper
  • Durable current term, voted-for and log, fsynced before replying, surviving restarts
  • A key-value store with get, put and compare-and-set, and a client library that handles leader redirection and retries without duplicate writes
  • Snapshotting and log compaction so the log does not grow without bound, with snapshot install for lagging followers
  • A fault injection harness that partitions the network, kills and restarts nodes, and delays, drops and reorders messages
  • Linearizability checking of recorded client histories with Porcupine across at least 1,000 randomised runs
  • A metrics dashboard for leader changes, commit latency and log size during fault tests
  • A design document explaining every safety argument and every bug the harness found

If you have time left

  • Membership changes allowing nodes to join and leave a running cluster
  • Lease-based or ReadIndex reads that avoid a full log append, with the safety trade-off documented
  • Sharding across several independent Raft groups with a simple shard controller

The problem: consensus is used everywhere and understood by few

Every modern infrastructure stack depends on consensus somewhere. Kubernetes stores cluster state in etcd, which runs Raft. Many databases and message queues use Raft or a close relative to agree on who is leader and what has been committed. Yet most graduates have only seen consensus on a slide, and cannot explain why a node must write its vote to disk before replying.

This project fixes that with an unambiguous test: the store either stays consistent under partitions and crashes, or it does not.

What already exists and the gap

  • etcd's raft library is a production Go implementation used by etcd and other systems.
  • HashiCorp raft is a Go library behind Consul and other HashiCorp tools.
  • MIT 6.5840 (formerly 6.824) has students build Raft and a fault-tolerant key-value store in Go in its labs.
  • Porcupine is an open source linearizability checker, originally built for the 6.5840 labs and now used in etcd's robustness tests.

Production libraries are too large to learn from and course labs come with a ready-made test harness. The gap this project fills is building your own fault injection harness and consistency checking around your own implementation, so the evidence of correctness is yours. If your university uses the MIT labs, do not copy lab solutions; this project deliberately replaces the provided harness with one you design.

What you build

  • Raft core: election, replication, commit, persistence.
  • Key-value service on top, with client sessions and request IDs so a retried put is applied once.
  • Snapshots and log compaction.
  • Harness: a controllable network layer between nodes that can partition, drop, delay and reorder messages, plus a scheduler that kills and restarts nodes.
  • Checker: record every client operation with start and end times and verify the history with Porcupine.
  • Dashboard: leader changes, commit latency and log size during tests.

Users: you and your examiners first, then students on a distributed systems course who can run the harness against their own implementation.

The technically interesting part

The test harness is half the project. Anyone can write code that works when nothing fails. The contribution is a harness that partitions the network, kills the leader mid-append, restarts nodes with stale logs, and then verifies the observed history is linearizable. Make every run reproducible from a seed so a failing run can be replayed and debugged.

Persistence is the part everyone gets wrong. Current term and voted-for must be durable before replying to a vote request. Skipping this produces a system that passes most tests and can elect two leaders in the same term after a restart. Show your supervisor a harness seed that reproduces this bug in a deliberately broken build, and the same seed passing on the correct one.

How to evaluate it

  • Safety: at least 1,000 randomised fault runs with zero linearizability violations; report every violation found during development and its root cause.
  • Liveness: time to elect a new leader after a leader crash, as a distribution over runs.
  • Performance: throughput and p99 commit latency for 3 and 5 node clusters, with and without batching.
  • Recovery: time for a node that was down for N entries to catch up, with and without snapshots.

Suggested plan

  1. Weeks 1 to 2: read the extended Raft paper twice; design the RPCs and the network layer.
  2. Weeks 3 to 5: election and replication with basic tests.
  3. Weeks 6 to 7: persistence, the key-value service, client sessions.
  4. Weeks 8 to 10: fault injection harness, Porcupine integration, bug hunting.
  5. Weeks 11 to 12: snapshots, metrics, performance runs.
  6. Weeks 13 to 14: design document and demo.

Path to a product

Do not force a SaaS pitch here. The realistic outcomes are: a portfolio centrepiece that opens interviews at infrastructure, database and cloud companies; an open source fault injection harness other learners can point at their own Raft; and a teaching resource your department's distributed systems course could adopt. The harness is the most reusable part, so package it cleanly.

Scope warning

Single Raft group. Sharding and membership changes are stretch goals. Do not build a SQL layer or a web admin panel. Implement exactly what the paper's Figure 2 says before optimising anything. A correct single group with a rigorous harness is a strong project on its own.

Learn what you need on Sefism

Get help with this idea

This is a demanding project, and teams that succeed plan the harness from week one. A mentor can help you split work between team members, set checkpoints and prepare to defend your safety arguments. If you are new here, create a free account first.

Ideas and guidance, not finished projects

These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.

Not sure this one fits you?

I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.

Discuss this idea
Work with me

Stuck on something specific?

Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.

Book a free sessionSee all services

Follow along

New writing, resources and project ideas land here first.