Production Raft clusters are not static. Nodes fail, get replaced, or are added to increase replication factor. Changing the set of voting members while the cluster continues to serve reads and writes is one of the most subtle problems in consensus. A careless membership change can create two independent majorities that both believe they are authoritative, permanently splitting the cluster’s state.
The Raft paper describes two approaches: single-server changes (adding or removing one node at a time) and joint consensus (transitioning through an intermediate configuration where both the old and new membership sets must agree). The etcd/raft library exposes both via ConfChange (single) and ConfChangeV2 (joint) APIs.
Understanding membership changes is essential for operating distributed databases at scale, where shard rebalancing, node decommissioning, and failure replacement all trigger reconfiguration. Getting it wrong doesn’t just cause downtime — it can cause undetectable data divergence that only surfaces weeks later during a consistency audit.
The fundamental safety invariant: at no point during a membership change can two disjoint majorities exist simultaneously. Single-server changes guarantee this by construction; joint consensus enforces it through a two-phase commit of the configuration.
Learners solve the “cold start” problem. A newly added voter that has no data would slow down the group (it can’t acknowledge entries it hasn’t received). By first adding it as a learner, the leader streams the full log/snapshot before promoting it, ensuring the new voter is immediately useful.
The operator (or automated controller) proposes a ConfChange entry to the leader. The leader appends it to the log and replicates it. Once committed, every node applies the configuration change — updating its peer list and quorum calculation. Because only one node changes at a time, the old majority and the new majority must share at least one node, preventing split brain.
For changes involving multiple nodes simultaneously (e.g., replacing two of three nodes), the leader first proposes a joint configuration C_old,new that requires agreement from majorities of both the old and new sets. Once this joint entry is committed, the leader proposes the final configuration C_new. After C_new commits, the transition is complete.
A safe node addition follows this sequence: (1) add the node as a learner via ConfChange, (2) wait for the learner to catch up to the leader’s log, (3) promote the learner to voter via another ConfChange. This prevents the new voter from causing availability disruptions while it bootstraps.
Configuration changes take effect immediately when the entry is appended to the log, not when it is committed. This means a leader may step down during a membership change if the new configuration no longer includes it. The Raft protocol handles this gracefully, but application code must be prepared for leadership transitions during reconfiguration.
// Go: Single-server membership change with etcd/raft
func (s *RaftServer) AddVoter(nodeID uint64, addr string) error {
// Step 1: Add as learner first
cc := raftpb.ConfChange{
Type: raftpb.ConfChangeAddLearnerNode,
NodeID: nodeID,
Context: []byte(addr),
}
if err := s.node.ProposeConfChange(context.TODO(), cc); err != nil {
return fmt.Errorf("propose add learner: %w", err)
}
// Step 2: Wait for learner to catch up
if err := s.waitForCatchUp(nodeID, 30*time.Second); err != nil {
return fmt.Errorf("learner catch-up: %w", err)
}
// Step 3: Promote to voter
cc = raftpb.ConfChange{
Type: raftpb.ConfChangeAddNode,
NodeID: nodeID,
}
if err := s.node.ProposeConfChange(context.TODO(), cc); err != nil {
return fmt.Errorf("propose promote voter: %w", err)
}
return nil
}
func (s *RaftServer) waitForCatchUp(nodeID uint64, timeout time.Duration) error {
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
status := s.node.Status()
if pr, ok := status.Progress[nodeID]; ok {
leaderLast := status.Commit
if pr.Match >= leaderLast-100 { // within 100 entries
return nil
}
}
time.Sleep(500 * time.Millisecond)
}
return fmt.Errorf("node %d did not catch up within %v", nodeID, timeout)
}// Go: Joint consensus for multi-node replacement
func (s *RaftServer) ReplaceNodes(removals, additions []uint64) error {
// Build a ConfChangeV2 with both removals and additions
var changes []raftpb.ConfChangeSingle
for _, id := range removals {
changes = append(changes, raftpb.ConfChangeSingle{
Type: raftpb.ConfChangeRemoveNode,
NodeID: id,
})
}
for _, id := range additions {
changes = append(changes, raftpb.ConfChangeSingle{
Type: raftpb.ConfChangeAddNode,
NodeID: id,
})
}
cc := raftpb.ConfChangeV2{
Changes: changes,
Transition: raftpb.ConfChangeTransitionJointExplicit,
}
// Phase 1: Enter joint consensus C_old,new
if err := s.node.ProposeConfChange(context.TODO(), cc); err != nil {
return fmt.Errorf("propose joint config: %w", err)
}
// Wait for joint config to commit
if err := s.waitForConfState(true); err != nil {
return fmt.Errorf("joint config commit: %w", err)
}
// Phase 2: Leave joint consensus, enter C_new
ccLeave := raftpb.ConfChangeV2{} // empty V2 signals "leave joint"
if err := s.node.ProposeConfChange(context.TODO(), ccLeave); err != nil {
return fmt.Errorf("propose leave joint: %w", err)
}
return nil
}Membership changes are a daily operational reality in distributed database systems at scale:
ConfChange for routine node additions and removals during shard rebalancing, with the learner workflow to prevent availability dips.ConfChangeV2 for multi-node changes.