Single-Leader, Multi-Leader, and Leaderless Replication

Distributed Systems advanced Building with AI ~38 min read

Overview

Replication is the act of keeping copies of the same data on multiple machines, but the hard design question is who is allowed to accept writes. DDIA chapter 5 frames most production database replication as one of three families: a single leader accepts all writes, multiple leaders accept writes in different places, or no leader exists and clients coordinate directly with several replicas. These are not implementation details. They determine write availability, conflict behavior, failover time, read freshness, and the shape of every operational incident.

Single-leader replication, used by systems such as PostgreSQL streaming replication, gives one primary the authority to order writes. Replicas follow by applying a log, often asynchronously. Multi-leader replication gives two or more primaries write authority, which is useful for disconnected clients and multi-region write locality but makes conflict detection unavoidable. Leaderless replication, popularized by Amazon Dynamo and inherited by Cassandra-style systems, lets clients or coordinators write to several replicas and use quorums, version vectors, read repair, and anti-entropy to converge.

The Dynamo paper is the canonical leaderless reference because it made the trade-off explicit: Amazon needed shopping carts and other key-value services to remain writable during disk failures, network flaps, and datacenter problems. The paper reports service-level thinking at the 99.9th percentile, including an example SLA of 300 ms for 99.9% of requests at 500 requests per second, and describes production services serving tens of millions of requests during peak shopping days. The lesson is not that every system should copy Dynamo. It is that replication topology is a product decision disguised as infrastructure.

Core Concepts

Single-Leader Replication

A single-leader system routes writes to one primary. The leader appends the write to its local log, applies it, and streams the log to followers. Followers can serve reads if the application tolerates replica lag, or they can remain purely standby nodes for failover. The main advantage is a simple ordering rule: if the leader accepted write A before write B, every follower should apply A before B.

The cost is a write bottleneck and a failover problem. If the leader is slow, unreachable, or isolated, writes either stop or must be redirected after promotion. Asynchronous followers can lose acknowledged writes during failover. Synchronous followers reduce data loss but add commit latency and can make the primary unavailable when not enough standbys acknowledge.

Multi-Leader Replication

Multi-leader replication allows more than one node to accept writes. The common deployment is multi-region: each region accepts local writes, then asynchronously exchanges changes with the others. This improves write latency for geographically distributed users and can continue operating during a temporary WAN break.

The price is concurrent update conflict. Two leaders can accept incompatible writes to the same record before learning about each other. CouchDB embraces this model by treating replication as incremental synchronization between databases and surfacing conflicted document revisions for the application to resolve. Multi-leader is therefore not simply “single-leader, but faster.” It moves correctness into conflict semantics.

Leaderless Replication

Leaderless systems remove the privileged primary. A client sends a write to several replicas, waits for enough acknowledgements, and considers the write accepted. Reads also consult several replicas and reconcile versions. The coordinator can be any node; if the natural replicas for a key are down, some systems use nearby healthy nodes temporarily and repair the intended replicas later.

Dynamo combined consistent hashing, preference lists, quorum-like reads and writes, vector clocks, sloppy quorum, hinted handoff, read repair, and Merkle-tree anti-entropy. Together, those mechanisms made writes highly available while admitting that clients might later see multiple sibling versions requiring application-assisted conflict resolution.

Key Concept: Write Authority Defines the System

The core replication question is not “how many copies exist?” but “who can decide the next valid value?” Single-leader systems centralize write authority, multi-leader systems partition or duplicate it, and leaderless systems distribute it across quorum responses and conflict-resolution rules.

How It Works

Single-Leader Write Path

The primary serializes writes into a log. Followers continuously fetch or receive that log and replay it in order. In PostgreSQL, asynchronous streaming replication does not make a commit wait for a standby by default. When synchronous replication is configured, settings such as synchronous_standby_names can require commits to wait for a priority standby or a quorum of named standbys.

function singleLeaderWrite(primary, followers, tx):
    lsn = primary.wal.append(tx)
    primary.apply(tx)

    if tx.synchronous:
        required = chooseStandbys(followers, tx.syncPolicy)
        waitUntilEvery(required, f => f.confirmedLsn >= lsn)

    return CommitAck(lsn)

function followerLoop(follower, primary):
    while true:
        records = primary.streamWal(after=follower.confirmedLsn)
        for record in records:
            follower.apply(record)
            follower.confirmedLsn = record.lsn

Multi-Leader Conflict Path

In a multi-leader topology, each leader must attach enough metadata to detect concurrent writes. A simple timestamp can pick a winner, but it can silently drop user intent when clocks disagree or two users edit different fields. Revision trees, vector clocks, CRDTs, or application-specific merge functions preserve more information at the cost of complexity.

function replicateFrom(peer, local):
    changes = peer.changesSince(local.checkpoint[peer.id])
    for change in changes:
        current = local.get(change.key)

        if current == null or descendsFrom(change.rev, current.rev):
            local.put(change.key, change.value, change.rev)
        else if descendsFrom(current.rev, change.rev):
            continue
        else:
            local.storeConflict(change.key, [current, change])

    local.checkpoint[peer.id] = changes.lastSequence

Leaderless Read and Write Path

In a leaderless system, the coordinator sends requests to the replica preference list for the key. The write succeeds after W acknowledgements; the read returns after R responses. If R + W > N, a read quorum and write quorum overlap in at least one replica under the simple no-concurrency model. Real systems still need version metadata because concurrent writes and failures break the simple intuition.

function leaderlessPut(key, value, context):
    replicas = preferenceList(key)
    version = incrementVectorClock(context.vectorClock, localNodeId)
    message = VersionedValue(value, version)

    acks = sendToReplicas(replicas, message)
    if countSuccessful(acks) >= W:
        return WriteAccepted(version)
    return WriteFailed(acks)

function leaderlessGet(key):
    replicas = preferenceList(key)
    responses = readFromReplicas(replicas)
    versions = waitForAtLeast(responses, R)
    latest = discardVersionsThatAreAncestors(versions)

    if len(latest) == 1:
        repairStaleReplicasAsync(key, latest[0], versions)
    return latest

Practical Applications

Relational Primary with Read Replicas

The standard web application deployment is single-leader: one PostgreSQL primary accepts writes, standbys serve analytics or read-heavy traffic, and failover tooling promotes a standby when the primary dies. This fits workloads that need SQL transactions, uniqueness constraints, and a clear source of truth. It also matches operational teams that prefer predictable conflict behavior over regional write availability.

Offline-First and Multi-Region Documents

Multi-leader replication fits disconnected collaboration and region-local writes. A field worker can edit a local database while offline, later syncing with a central server. A global document service can accept writes in Europe and North America without sending every edit through one distant primary. The application must still define what happens when two people edit the same logical object concurrently.

Always-Writable Key-Value Services

Leaderless replication fits shopping carts, session stores, preference stores, and other primary-key workloads where rejecting writes is worse than asking the application to merge versions later. Dynamo explicitly targeted small objects, usually less than 1 MB, and avoided multi-key transactions. That scope made the conflict model manageable enough for high availability to be worth the trade.

Implementation Patterns

Choose Per-Key Ownership Before Choosing Product Names

Do not start with “Postgres vs Cassandra.” Start with the ownership rule for a single key: one writer, region-local writers, or quorum-mediated writers. Product choices become much clearer once the application has stated whether conflicts are bugs, rare repair events, or a normal part of the API.

type ReplicationChoice struct {
    Topology         string
    WriteAuthority   string
    ConflictPolicy   string
    TypicalUseCase   string
}

var choices = []ReplicationChoice{
    {
        Topology:       "single-leader",
        WriteAuthority: "one primary per shard",
        ConflictPolicy: "avoid by serializing writes",
        TypicalUseCase: "SQL OLTP, metadata stores, ordered logs",
    },
    {
        Topology:       "multi-leader",
        WriteAuthority: "several primaries",
        ConflictPolicy: "detect and merge concurrent revisions",
        TypicalUseCase: "offline-first apps, multi-region documents",
    },
    {
        Topology:       "leaderless",
        WriteAuthority: "quorum of replicas",
        ConflictPolicy: "return siblings or resolve by version rules",
        TypicalUseCase: "always-writable key-value stores",
    },
}

Expose Consistency in the Client Contract

Replication choices leak into user-visible behavior. A read replica may be stale. A multi-leader document may return conflicts. A leaderless read may return siblings. Production APIs should name those outcomes instead of pretending all reads are fresh and all writes are final. This is especially important during incident response, when operators need to know whether they are looking at lag, conflict, or data loss.

Performance Considerations

Common Pitfalls

Further Reading