Dynamo-style databases replicate each key to several nodes, then let reads and writes choose how many replicas must answer before the operation succeeds. This is the tunable consistency model behind systems such as Dynamo, Cassandra, Riak, ScyllaDB, and Amazon Keyspaces. The vocabulary is compact: N is the replication factor, W is the number of write acknowledgements required, and R is the number of read responses required.
The famous rule is W + R > N. With three replicas, a write quorum of two and a read quorum of two overlap in at least one replica, so a read should see the latest successful write under a simplified model with no concurrent writes, no clock anomalies, and no failed partial writes. Real systems need more machinery. Replicas can be down. Coordinators can time out after only some replicas apply a mutation. Reads can discover stale versions. Concurrent writes can produce siblings. Sloppy quorum, hinted handoff, read repair, and full repair exist to make that messy world converge.
The operational point is availability control. A database can accept a write after one replica, two replicas, or all replicas acknowledge; each choice changes latency, durability confidence, and failure behavior. Dynamo was designed for services where rejecting a shopping-cart update during failures was worse than returning multiple versions later. Cassandra and ScyllaDB expose similar ideas as per-operation consistency levels and anti-entropy processes.
A quorum is any subset large enough to overlap with another subset of interest. In a replicated key-value store with N=3, the usual quorum size is two. If a write reaches two replicas and a read checks two replicas, at least one replica should be common to both sets. The coordinator can use version metadata to choose the newest value and repair stale replicas.
Quorums are tunable, not magical. W=1, R=1 maximizes availability and minimizes latency but can return stale data. W=3, R=1 makes writes wait for every replica and reads cheap. W=1, R=3 makes writes cheap and reads expensive. W=2, R=2 is the common balanced profile for three replicas.
A strict quorum only counts acknowledgements from the key’s designated replicas. A sloppy quorum counts healthy fallback nodes when designated replicas are unreachable. This keeps the system writable during failures, but the write is now temporarily stored outside its normal home. Sloppy quorum improves availability by weakening the guarantee that the quorum overlaps the exact replica set a later read will consult.
Hinted handoff is the repair path for sloppy or partially failed writes. A coordinator stores a hint containing the target replica identity and mutation data, then replays that hint when the target node comes back. ScyllaDB documents this as a hint equal to target replica ID plus mutation data; Cassandra describes hints as one of the mechanisms, alongside read repair and anti-entropy repair, that help eventually deliver updates to all replicas.
Read repair uses reads to discover divergent replicas. The coordinator asks replicas for data or digests, compares versions, returns the winning value to the client, and sends repair writes to stale replicas. Full repair or anti-entropy scans data ranges in the background so that keys not recently read still converge.
R and W are not just consistency knobs. They decide how many machines sit on the critical path. A quorum write is only as fast as the slowest replica required for acknowledgement, and a quorum read is only as available as enough replicas in the chosen consistency level.
The coordinator maps a key to a preference list, sends the mutation to the first N replicas, and waits for W acknowledgements. Reads wait for R responses and compare versions. In production, the coordinator should continue sending to all replicas even after the client-visible threshold is met, because the extra responses reduce repair work later.
function quorumWrite(key, value, requiredW):
replicas = naturalReplicas(key) // size N
version = nextVersion(key)
pending = []
for replica in replicas:
pending.append(sendAsync(replica, Put(key, value, version)))
acks = waitForSuccesses(pending, requiredW)
if len(acks) >= requiredW:
return Accepted(version)
return Failed("not enough replica acknowledgements")
function quorumRead(key, requiredR):
replicas = naturalReplicas(key)
responses = waitForSuccesses(readAsync(replicas, key), requiredR)
winner = chooseByVersion(responses)
stale = [r for r in responses if r.version < winner.version]
scheduleReadRepair(key, winner, stale)
return winner.valueSloppy quorum extends the write path. If a natural replica is down, the coordinator sends the write to the next healthy node and records where the mutation really belongs. The fallback node must not become the permanent owner of the key; it is holding a repair obligation.
function sloppyWrite(key, value, requiredW):
natural = naturalReplicas(key)
candidates = preferenceListIncludingFallbacks(key)
version = nextVersion(key)
acks = []
for node in candidates:
if len(acks) == requiredW:
break
target = firstUnavailableNaturalReplicaNotCovered(natural, acks)
if node in natural:
ok = node.put(key, value, version)
else:
hint = Hint(targetReplica=target.id, key=key, value=value, version=version)
ok = node.storeHintAndMutation(hint)
if ok:
acks.append(node)
if len(acks) >= requiredW:
return Accepted(version)
return Failed("not enough healthy nodes")Read repair must preserve the system’s conflict semantics. If one version dominates all others, stale replicas can be updated automatically. If there are concurrent versions, the read path may return siblings to the application rather than inventing a destructive winner.
function reconcileRead(key, versions):
maximal = discardDominatedVersions(versions)
if len(maximal) == 1:
winner = maximal[0]
for replicaVersion in versions:
if replicaVersion.version < winner.version:
asyncPut(replicaVersion.replica, key, winner.value, winner.version)
return ReadResult(value=winner.value)
return ReadResult(conflicts=maximal)Cassandra-family systems expose quorum behavior as consistency levels such as ONE, QUORUM, LOCAL_QUORUM, and ALL. With replication factor three, QUORUM means two replicas. LOCAL_QUORUM keeps the quorum inside the local datacenter to avoid paying a cross-region round trip on every operation.
Dynamo targeted services where availability was part of the user experience. Shopping carts, sessions, preferences, and feature state often prefer accepting a write during partial failure and merging later. These workloads also fit the key-value model: small objects, no cross-key transactions, and application code capable of resolving divergent versions.
Hinted handoff is most visible during node restarts, network breaks, and overload. If a node is down briefly, hints can replay missed writes. If it is down longer than the hint window or hints are lost, operators must run repair. ScyllaDB documents a default maximum hint window of three hours, which is a concrete reminder that hints are a short-term availability tool rather than a replacement for repair.
Treat consistency level as part of the operation, not as a hidden global. Authentication reads, payment state, and user-visible profile updates may need different policies than metrics, caches, or append-only activity events. A clean client makes that choice obvious at the call site.
type Consistency int
const (
One Consistency = iota
Quorum
All
)
func requiredResponses(level Consistency, replicationFactor int) int {
switch level {
case One:
return 1
case Quorum:
return replicationFactor/2 + 1
case All:
return replicationFactor
default:
panic("unknown consistency level")
}
}
func GetUser(ctx context.Context, key string, level Consistency) (User, error) {
r := requiredResponses(level, 3)
result := quorumReadWithContext(ctx, key, r)
if len(result.Conflicts) > 0 {
return User{}, ConflictError{Versions: result.Conflicts}
}
return decodeUser(result.Value)
}A node that stores hints is not a new replica owner. Keep hinted data in a separate queue with target replica identity, replay deadline, backpressure, metrics, and deletion after successful handoff. Mixing hints into normal storage without metadata makes rebalancing and repair ambiguous.
A high write success rate can hide growing replica divergence. Track hint queue depth, hint replay age, read repair counts, digest mismatches, repair duration, and percent of token ranges repaired within policy. These metrics tell you whether the system is merely accepting writes or actually converging.
N=3, balanced quorum is W=2, R=2. For N=5, a common quorum is three. Larger N improves failure tolerance but increases storage cost and the fan-out of every operation.ALL gives the strongest acknowledgement rule but any unavailable replica can fail the operation. ONE maximizes availability but can read stale data and increases dependence on repair.max_hint_window_in_ms of three hours. If a replica is down beyond the window, new hints are not generated until it returns, so full repair becomes mandatory for convergence.W + R > N means linearizability: The overlap rule does not handle concurrent writes, last-write-wins timestamp anomalies, sloppy quorum fallback nodes, or failed writes that reached only a minority before timing out.QUORUM and immediately reads at ONE can hit a stale replica. Use session stickiness, higher read consistency, or explicit version checks for user-visible confirmation screens.