Distributed systems must survive crashes without losing committed data. Snapshots provide point-in-time captures of a node’s entire state machine, enabling fast recovery without replaying the full write-ahead log from genesis. Combined with incremental log replay from the snapshot index forward, they bound recovery time to seconds instead of hours.
In Raft-based systems, snapshots serve a dual purpose: they compact the log (freeing disk and memory) and provide a fast bootstrap mechanism for new or lagging replicas. A leader can ship a snapshot to a follower that is too far behind, avoiding the need to retain unbounded log history.
Getting snapshots right is deceptively hard. The snapshot must be consistent (capturing all mutations up to a specific log index), durable (surviving partial writes via atomic directory swaps), and verifiable (using CRC checksums to detect corruption). A single bug in any of these areas can silently lose committed data or leave a node in an irrecoverable state.
A snapshot is only useful if it is consistent — it must reflect the state machine after applying all entries up to index N and none beyond N. Partial or interleaved state produces silent data corruption on restore.
Atomic directory swaps (via rename(2)) are the standard technique for crash-safe snapshot installation. The old snapshot remains intact until the rename succeeds, so a crash mid-write leaves the previous good snapshot in place.
The state machine serializes its in-memory state at a specific applied index. In a typical Raft state machine implementation, this involves iterating over all key-value pairs in the store and writing them to a structured file format. The process must either hold a read lock or operate on an immutable copy to avoid capturing partial writes from concurrent mutations.
The serialized data is written to a temporary directory (e.g., snapshot-tmp-{id}/). A CRC32 checksum is computed over the entire payload and written as a metadata file. Only after both the data and metadata are fully fsynced does the system rename the temporary directory to the canonical snapshot path.
On startup, the node locates the latest valid snapshot by scanning the snapshot directory, verifying each candidate’s CRC. It deserializes the snapshot into the state machine, sets the applied index, and then replays any log entries after the snapshot index. This two-phase restore ensures both consistency and minimal replay time.
Always fsync both the snapshot data and the parent directory before performing the atomic rename. Without this, a power failure can leave the directory entry pointing to unflushed data, producing a corrupt snapshot that passes the rename check but fails CRC validation.
Most systems trigger snapshots based on log size thresholds (e.g., every 10,000 entries) or time intervals. The trigger runs on a background goroutine to avoid blocking the main apply loop. A mutex or channel-based gate prevents concurrent snapshot operations.
When a leader detects that a follower needs entries that have been truncated, it initiates an InstallSnapshot RPC. The snapshot is streamed in chunks to bound memory usage on both sides. The follower writes each chunk, verifies the final CRC, and atomically installs the result.
// Go: Snapshot creation with atomic swap
func (s *Store) TakeSnapshot(appliedIndex uint64) error {
tmpDir := fmt.Sprintf("%s/snapshot-tmp-%d", s.snapshotDir, appliedIndex)
if err := os.MkdirAll(tmpDir, 0755); err != nil {
return fmt.Errorf("create tmp dir: %w", err)
}
// Serialize state under read lock
s.mu.RLock()
data, err := s.serializeState()
s.mu.RUnlock()
if err != nil {
return fmt.Errorf("serialize: %w", err)
}
// Write data file
dataPath := filepath.Join(tmpDir, "state.bin")
if err := writeFileSync(dataPath, data); err != nil {
return fmt.Errorf("write state: %w", err)
}
// Write CRC metadata
crc := crc32.ChecksumIEEE(data)
meta := SnapshotMeta{Index: appliedIndex, CRC: crc}
metaBytes, _ := json.Marshal(meta)
metaPath := filepath.Join(tmpDir, "meta.json")
if err := writeFileSync(metaPath, metaBytes); err != nil {
return fmt.Errorf("write meta: %w", err)
}
// Atomic swap
finalDir := fmt.Sprintf("%s/snapshot-%d", s.snapshotDir, appliedIndex)
return os.Rename(tmpDir, finalDir)
}// Go: Recovery from latest snapshot
func (s *Store) Restore() error {
snap, err := s.findLatestValidSnapshot()
if err != nil {
return fmt.Errorf("find snapshot: %w", err)
}
if snap == nil {
log.Println("no snapshot found, replaying full log")
return s.replayLog(0)
}
// Verify CRC
data, err := os.ReadFile(filepath.Join(snap.Dir, "state.bin"))
if err != nil {
return fmt.Errorf("read snapshot: %w", err)
}
if crc32.ChecksumIEEE(data) != snap.Meta.CRC {
return fmt.Errorf("snapshot CRC mismatch at index %d", snap.Meta.Index)
}
// Deserialize into state machine
if err := s.deserializeState(data); err != nil {
return fmt.Errorf("deserialize: %w", err)
}
s.appliedIndex = snap.Meta.Index
// Replay log entries after snapshot
return s.replayLog(snap.Meta.Index + 1)
}In a production distributed database, snapshots are used across multiple components:
fsync calls on both the file and directory, a power failure can corrupt the snapshot even though the application completed successfully.rename(2) ensure crash-safe snapshot installation.