the problem
Every engineer has a bug that happens once a week in production and never in testing.
This turns it into a seed number.
The non-determinism that makes a concurrency bug unreproducible does not come from your program. It comes from the OS scheduler, the clock and the network. Replace all three with things you control, drive them from a single integer, and the program becomes a pure function of that integer — so a failure stops being an intermittent issue and becomes a permanent address.
loading stored runs…
What actually happened, step by step
Five processes on five channels; simulated time runs left to right. Each arrow is one message. Gold marks a fault the network injected on purpose — a drop, a delay, a duplicate. The crimson rule is the instant an invariant stopped being true.
- message
- drop
- delay
- duplicate
- timeout
- invariant violated
- click an event, or focus the chart and use ← → Home End V Space
Then it deletes everything that was not the bug
A failing seed arrives as a wall of events. Delta-debugging re-runs the scenario with faults and workload removed, keeping only what still reproduces the same named invariant failure. Below: every event of the original run, and what survived.
The fix that looks exactly like a fix
Three versions of the same consumer, 10,000 seeds each, on a network that drops, delays and duplicates. On a perfect network all three pass 10,000 out of 10,000 — which is what a normal test suite measures.
Why the seed numbers can be trusted
What it cannot do
-
Interleavings are explored only at yield points. A process runs
uninterrupted between one
yieldand the next, so a true data race on shared memory is invisible to this tool by construction. It is the right tool for message passing, timeouts, retries and idempotency; it is the wrong tool for memory races. - Only what you model gets tested. The scheduler explores the generators you wrote, not your production service. If the model and the service disagree, the model wins and you have verified fiction.
- Shrinking finds something smaller, not something minimal. Removing a fault changes the message sequence, so shrinking is a search over configurations. It took seed 1 from 91 events to 43; it took seed 11 from 141 to 139. The report says which.
- The determinism guard is a Python-level patch, not a sandbox. Code holding a pre-bound reference slips through, as does anything inside a C extension. It raises the cost of a mistake from zero to high; it does not make one impossible.