Raft Liveness Under Partial Partitions

Overview

Raft is usually introduced as a consensus algorithm that stays available as long as a majority of servers can communicate. That statement is broadly true for crash failures, but it is incomplete for partial omission faults: network patterns where some nodes can talk to some peers but not others. Under these conditions, a cluster may retain a healthy communicating majority and still fail to hold a stable leader.

The Decentralized Thoughts analysis shows why. A partially connected follower that cannot hear from the leader can repeatedly time out, increment term, and trigger other servers to step down even though a majority could otherwise keep serving. The result is not a split-brain safety failure, but a liveness failure: the cluster thrashes through elections and cannot make forward progress.

The practical fix is not “just enable PreVote.” PreVote addresses one failure mode by preventing isolated followers from bumping term unless they can plausibly win an election, but the same article shows a second scenario where PreVote alone still deadlocks. The missing piece is CheckQuorum: leaders must step down when they are no longer receiving responses from a majority.

Together, PreVote + CheckQuorum form an operational liveness package. PreVote stops disconnected minority nodes from needlessly disrupting healthy leaders, and CheckQuorum stops stranded leaders from indefinitely blocking the election of a reachable replacement.

Core Concepts

Safety vs. Liveness

Safety means the system never commits conflicting histories. Liveness means the system eventually elects a leader and commits new work. Raft’s safety properties can hold while liveness fails under asymmetric connectivity: no two leaders commit conflicting logs, but no stable leader remains long enough to make progress.

PreVote

PreVote adds a trial election phase before term is incremented. A would-be candidate asks peers whether it would receive votes. If it cannot gather a majority, it does not disturb the cluster by moving to a higher term. This protects healthy leaders from disruption by isolated followers or removed nodes that can talk to only a small subset of peers.

CheckQuorum

CheckQuorum makes leadership conditional on recent majority activity. If the leader stops hearing back from a quorum for an election timeout, it steps down. This matters because a leader with only minority reachability can otherwise keep enough followers from supporting new elections to stall the cluster indefinitely.

How It Works

Failure Mode 1: Minority Followers Repeatedly Disrupt the Leader

In the Decentralized Thoughts example, servers 1, 2, and 3 are mutually connected, while server 4 talks only to 2 and server 5 talks only to 3. Without PreVote, servers 4 and 5 time out because they do not receive heartbeats from the leader, increment term, and send vote requests into the majority component. Those term bumps force the current leader to step down, so the majority repeatedly loses leadership even though it is internally healthy.

Failure Mode 2: PreVote Alone Can Still Deadlock

PreVote fixes the first case, but not the second. If an old leader remains connected to one follower yet disconnected from the majority, that follower may continue hearing heartbeats and refuse to pre-vote for a reachable replacement. The old leader cannot commit because it lacks a quorum, but the new majority cannot elect either. Liveness is still broken.

c := &raft.Config{
    ID:              0x01,
    ElectionTick:    10,
    HeartbeatTick:   1,
    Storage:         storage,
    MaxSizePerMsg:   4096,
    MaxInflightMsgs: 256,
    PreVote:         true,
    CheckQuorum:     true,
}

n := raft.StartNode(c, peers)

etcd/raft exposes both mitigations directly in configuration. The library documentation states that PreVote enables the pre-election algorithm from Ongaro’s thesis and that CheckQuorum causes the leader to step down when quorum is not active for an election timeout.

Leader Self-Demotion Under Quorum Loss

Internally, the etcd raft implementation periodically checks quorum activity. When that check fails, the leader reverts to follower instead of clinging to leadership. That makes room for a reachable server inside the majority-connected component to complete PreVote, win the real election, and resume progress.

Practical Applications

Control Planes and Metadata Stores

Systems such as etcd-backed control planes, service registries, and shard managers operate on imperfect cloud networks. Partial switch failures, asymmetric ACL problems, and one-way packet loss create exactly the omission-fault patterns discussed in the article. Operators who only test full-node crashes often miss these liveness hazards until production.

Multi-Raft Storage Systems

In multi-Raft databases, one flaky host can participate in many groups at once. If the raft implementation allows stranded leaders or disruptive minority followers, the blast radius multiplies across shards. Enabling PreVote and quorum checking at the library level prevents the same network pathology from causing system-wide election churn.

Implementation Patterns

Enable Both Options Together

The main implementation pattern is simple: treat PreVote and CheckQuorum as a pair, not as independent tuning extras. etcd/raft documents them separately, but the Decentralized Thoughts analysis makes their dependency clear from a liveness perspective.

for {
    select {
    case <-ticker.C:
        n.Tick()
    case rd := <-n.Ready():
        saveToStorage(rd.HardState, rd.Entries, rd.Snapshot)
        send(rd.Messages)
        for _, entry := range rd.CommittedEntries {
            apply(entry)
        }
        n.Advance()
    }
}

The operational implication is that you must keep the ready loop healthy as well. If message sends or heartbeat processing are delayed by overloaded disks, blocked networking, or a frozen event loop, a leader can falsely appear quorum-inactive and step down. Liveness features still depend on a timely implementation.

Test Asymmetric Network Scenarios Explicitly

Do not limit chaos tests to “kill one node” or “drop all traffic to one node.” Model directed packet loss and partial reachability: one follower hearing from the leader while the majority cannot, minority nodes reaching only one majority peer, or old leaders retaining one-way visibility. Those are the cases that distinguish plain Raft, PreVote-only Raft, and PreVote+CheckQuorum Raft.

Performance Considerations

• Election timeout tuning: Timeouts that are too short amplify transient packet loss into needless elections; too long delays recovery from a genuinely stranded leader.
• Heartbeat handling latency: CheckQuorum depends on timely response tracking. Slow storage, overloaded goroutines, or blocked transport paths can look like quorum loss.
• Shard multiplication: In multi-Raft systems, every extra election storm is multiplied across groups, so liveness bugs quickly become tail-latency and availability incidents.
• Read semantics: etcd/raft notes that lease-based read paths require CheckQuorum, because stale leaders must not continue serving as if leadership were proven.

Common Pitfalls

• Equating majority connectivity with automatic availability: A healthy majority is not sufficient if minority nodes can still cause term churn or a stranded leader can block elections.
• Enabling PreVote alone and assuming the problem is solved: PreVote removes one disruption path but leaves the stranded-leader deadlock unresolved.
• Ignoring asymmetric faults in testing: Full disconnection tests miss the exact reachability patterns that trigger these liveness failures.
• Treating leadership as binary instead of evidence-based: With CheckQuorum, leadership is not “I was elected once” but “I am still hearing from a majority now.”