Systems incident simulator
Mission Control
Break a system. Understand the trade-offs.
Commerce platform
Baseline- Root cause
- Impacted by a failing dependency
- Line colour: load vs. capacity
healthy · Cache hit 94%
Protects the read path. Coordinated expirations can send a burst to the database.
Event timeline
1Simulation time, not wall-clock time. Latest 8 actions and state transitions.
- Simulation ready at baseline.
Engineering debrief
Many callers miss the same cached value together. Spread expirations with TTL jitter, then coalesce identical in-flight requests so only one caller refills the key. Even then the hit rate climbs back over tens of seconds while the cache refills. The database is the loudest alarm, but the cache is the root cause.
The trade-off
Jitter changes freshness timing; coalescing adds coordination. More API replicas alone cannot expand the database’s fixed read budget.
Model boundaries
These are authored examples, not accounts of client incidents. Traffic, capacity and thresholds are deliberately simplified. Latency follows a queueing curve (ρ/(1−ρ)), so headroom matters well before 100% load. No live services are contacted.