Chaos Engineering Toolkit for Teaching Distributed Systems

The problem with teaching distributed systems failures

Distributed systems courses teach partitions, timeouts, split brain and retry storms as slides and diagrams. Students can recite the CAP theorem without ever having watched a replicated service accept conflicting writes during a partition, or seen a missing timeout turn one slow node into a stalled cluster. The lesson rarely sticks because it is never felt.

Instructors know this, but setting up a safe way to break things for a class of 60 students is a lot of work. Industry tools assume an operations team, not a lab session, and a fault left behind on a shared machine ruins the next group's work.

What already exists and the gap

• Chaos Monkey from Netflix randomly terminates instances in production so engineers build services that survive instance loss. It is the origin of the practice, but it only kills instances.
• Chaos Mesh is an open source, cloud native chaos platform that injects faults into Kubernetes workloads, nodes, network and cloud APIs.
• LitmusChaos, a CNCF incubating project, offers a hub of prebuilt experiments and probes that check a steady state hypothesis during a run.
• Toxiproxy from Shopify is a TCP proxy for simulating latency and dropped connections in tests.

These are built for engineers running production systems. None of them ship a small sample system, lab handouts, or a view that explains to a student what just happened and why. The gap is a teaching tool: fewer fault types, a guaranteed clean-up story you can trust in a lab, and experiments written as lessons. The same idea sits behind the labs in MIT's distributed systems course, where students test their Raft code against an unreliable simulated network.

What you build

Three roles:

• Instructor: writes or picks experiments, deploys the sample app for a lab group and reviews results.
• Student: runs an experiment, writes a hypothesis first, then compares it with what the graphs show.
• Node agent: a small privileged process on each node that applies and removes faults with tc netem, iptables rules, signals and stress processes.

The main flow: pick an experiment, state the hypothesis (for example "reads keep working during a partition, writes to the minority side fail"), start it, watch live metrics, and get a report that marks where reality disagreed with the hypothesis.

The technically interesting part

Recovery must be guaranteed even if your tool dies. If the controller crashes after partitioning two nodes, the partition must still end. Design every fault as a lease: the controller asks the node agent to apply a fault with a deadline, and the agent removes it when the deadline passes unless the lease is renewed. The agent also restores a clean state on startup, so a rebooted node cannot keep stale iptables rules. Show your supervisor the kill tests: controller killed mid-fault, agent restarted mid-fault, network cut between controller and agent.

The second interesting part is the hypothesis discipline. An experiment without a written expectation is vandalism. Requiring the hypothesis up front and scoring it against metrics afterwards is what makes the tool teach something.

How to evaluate it

• Safety: run at least 50 injected controller crashes across fault types and report how many left residue after the lease deadline. The target is zero.
• Fault accuracy: measure injected latency and loss with ping and a load test from k6, and report the error against what was requested.
• Teaching value: run a lab with 10 or more students. Give a short quiz on failure behaviour before and after, and collect a usability score.
• Instructor effort: time how long it takes a teaching assistant to set up a lab for one group from scratch.

Suggested plan

1. Weeks 1 to 2: study Chaos Mesh and Toxiproxy, choose the fault list, build the sample app with metrics.
2. Weeks 3 to 6: node agent with network, process and resource faults, plus the lease mechanism.
3. Weeks 7 to 9: controller, YAML experiments, hypothesis scoring and the results view.
4. Weeks 10 to 12: write six lab experiments with instructor notes; run the kill test suite.
5. Weeks 13 to 15: pilot lab with students, measure, write up.

Path to a product

The honest first user is your own department's distributed systems or cloud computing course. Publish the toolkit as open source with the lab pack, then share it with instructors at other universities. Adoption by a few courses is a strong portfolio story for infrastructure and SRE roles. A later option is a hosted lab environment sold per course or to bootcamps and corporate training teams, where setting up clusters is the painful part.

Scope warning

Container level faults on one orchestrator only. Do not build kernel level fault injection, multi-cloud support, or a scheduler for production use. Do not write your own metrics store; Prometheus is enough. Six excellent experiments beat twenty shallow ones.

Learn what you need on Sefism

• System design roadmap : replication, consistency and failure modes your experiments will demonstrate.
• DevOps and cloud cheat sheet : containers, Kubernetes and monitoring basics in one place.
• All Things Distributed : Werner Vogels on designing for failure at scale.
• Grafana k6 Documentation : generate steady load so faults show up in the graphs.
• roadmap.sh: Linux : networking and process tools the node agent relies on.

Get help with this idea

The risk in this project is spending the semester on fault types and leaving no time for the lease safety work and the lab pilot. Tauseef can help you cut the fault list, pick a sample app, and design an evaluation that shows both safety and teaching value. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session