Distributed systems often work in clean integration tests and fail under asymmetric partitions, process pauses, clock skew, packet loss, disk faults, and unlucky operation interleavings. Jepsen-style testing attacks those assumptions by running real systems on multiple nodes, generating concurrent client operations, injecting faults during the workload, and checking whether the observed operation history satisfies a stated correctness model.
The reusable pattern is not tied to one language or framework. It is a test architecture: a control node provisions database nodes, clients apply operations, a generator schedules work and faults, a nemesis disrupts the system, a history records operation invocations and completions, and checkers analyze the history for safety and sometimes availability or performance.
This differs from deterministic simulation testing. Simulation controls an in-memory model of time, network, and storage. Jepsen-style tests exercise deployed software through real network, process, operating-system, and storage boundaries. That makes them slower and less exhaustive, but much closer to production failure modes and deployment mistakes.
A history records every operation as an invocation and completion with process identity, operation type, value, outcome, and timing. Checkers interpret that history against a model such as linearizability, sequential consistency, serializability, read-your-writes, set semantics, or queue semantics.
Generators decide which operations each logical process should attempt and when. Clients translate abstract operations into real API calls against the system under test. This separation lets you change workloads without rewriting the protocol adapter.
A nemesis injects faults while clients are running. Common nemeses partition networks, kill processes, pause nodes, skew clocks, fill disks, restart services, drop traffic in one direction, or combine faults in schedules designed to stress consensus and replication assumptions.
Fault injection without a checker is only chaos. A Jepsen-style test must define what behavior is legal, record enough history to evaluate it, and fail with evidence when the system violates the model.
The control program connects to database nodes, installs or configures the system, starts services, launches logical client processes, schedules nemesis actions, collects histories, tears everything down, and writes test artifacts for review. The control node should be able to reproduce a run from parameters such as node count, operation mix, fault schedule, and random seed.
type Operation = {
process: number;
type: "invoke" | "ok" | "fail" | "info";
f: "read" | "write" | "cas";
value?: unknown;
timeNanos: number;
};
async function runFaultTest(cluster, workload, nemesis, checker) {
await cluster.install();
await cluster.start();
const history = new History();
const clients = workload.clients.map(client =>
runClientLoop(client, workload.generator, history)
);
const faultLoop = runNemesisSchedule(nemesis, history);
await sleep(workload.durationMs);
await Promise.allSettled([...clients, faultLoop.stop()]);
await cluster.teardown();
return checker.check(history.operations());
}A checker turns observed operations into evidence. For a register, it might search for a linearization order that respects real-time precedence. For a set, it may verify that every successful read is compatible with completed adds and removes. For a transaction system, it may build dependency graphs and detect cycles that violate the target isolation level.
Leader election, quorum writes, membership changes, lease reads, lock services, and replicated metadata stores are prime targets. Fault injection exposes cases where leaders serve stale reads, terms advance unexpectedly, or reconfiguration allows two groups to accept conflicting writes.
Jepsen-style tests can validate transaction isolation, compare-and-set registers, counters, sets, queues, streams, and job schedulers. Each data type needs a model-specific checker that understands which histories are legal.
Vector databases, feature stores, distributed caches, workflow engines, and model-serving control planes often depend on replicated state. Fault tests can verify that acknowledged index writes survive failover, routing tables converge, and background workers do not execute jobs twice after partitions.
Begin with a single register, set, queue, or transaction pattern. A narrow model gives you a checker that can produce concrete counterexamples instead of vague availability graphs. Once the basic model holds, add richer operations and harder faults.
function checkReadYourWrites(history) {
const lastWriteByProcess = new Map();
const errors = [];
for (const op of history) {
if (op.type !== "ok") continue;
if (op.f === "write") {
lastWriteByProcess.set(op.process, op.value);
}
if (op.f === "read" && lastWriteByProcess.has(op.process)) {
const expected = lastWriteByProcess.get(op.process);
if (!visible(op.value, expected)) {
errors.push({op, expected});
}
}
}
return {valid: errors.length === 0, errors};
}Record every nemesis action in the history. A failing run is only useful if the report shows which partition, pause, restart, or clock skew overlapped the bad operation. Treat nemesis events as first-class operations rather than out-of-band log messages.