A distributed lock is not a local mutex with a network call in front of it. The lock holder can pause, the network can delay packets, the lock service can expire a lease, and another client can acquire the same logical lock while the first client is still alive. If the protected resource accepts writes from both clients, the lock has failed even if the lock service behaved as designed.
Lease-based locking adds a timeout so abandoned locks eventually release. That helps liveness, but it weakens safety unless every operation under the lock is fenced. A fencing token is a monotonically increasing number issued when a client acquires the lease. The protected resource stores the highest token it has accepted and rejects any request with an older token.
Kleppmann’s Redlock critique made this pattern widely known: a lock service alone cannot stop a paused client from resuming after its lease expired and writing stale data. Fencing moves the final safety check to the resource being protected. The lock grants authority, but the resource verifies that the authority is still newer than any authority it has already seen.
This design is the standard answer for correctness-critical distributed locking. Use approximate locks for efficiency-only deduplication. Use consensus-backed leases plus fencing tokens for exclusive access that protects data integrity.
A lease is a lock with an expiration time. The client must keep it alive or renew it before expiry. Leases prevent permanent deadlock after client failure, but they rely on timing for ownership transfer. A slow client can continue executing after the lock service has already considered the lease expired.
A fencing token is a strictly increasing value assigned to each successful acquisition. If client A gets token 33, pauses, loses its lease, and client B gets token 34, the storage system can reject A’s later write because token 33 is older than the highest accepted token.
The token must be checked by the database, file service, payment processor, job table, or external system being protected. If only the client checks the token, a paused or buggy client can still write. The resource needs an atomic compare-and-update around the highest accepted token and the protected mutation.
A lease says who the coordinator currently believes owns the lock. A fencing token lets the protected resource reject stale owners. Correctness depends on the second property, not merely on the first.
The lock service must be able to generate monotonically increasing tokens. ZooKeeper can use zxid or znode version information. etcd exposes creation revisions for election leader keys and lock keys that can be used in transactions. A database can use a sequence inside a serializable transaction.
type LeaseGrant struct {
LockName string
Owner string
Token int64
Expires time.Time
}
func AcquireLease(tx Tx, name, owner string, ttl time.Duration) (LeaseGrant, error) {
row := tx.GetLockForUpdate(name)
now := tx.Now()
if row.Owner != "" && row.Expires.After(now) {
return LeaseGrant{}, ErrLocked
}
row.Token++
row.Owner = owner
row.Expires = now.Add(ttl)
tx.Save(row)
return LeaseGrant{Name: name, Owner: owner, Token: row.Token, Expires: row.Expires}, nil
}The protected table or service stores the highest token accepted for each resource. The update succeeds only when the incoming token is greater than or equal to the stored token and the mutation is applied in the same atomic operation.
func WriteWithFence(tx Tx, resourceID string, token int64, payload []byte) error {
resource := tx.GetResourceForUpdate(resourceID)
if token < resource.HighestFenceToken {
return ErrStaleLease
}
resource.HighestFenceToken = token
resource.Payload = payload
resource.UpdatedAt = tx.Now()
tx.Save(resource)
return nil
}Clients should still stop work when they detect lease loss, but that is an optimization and cleanup path. The safety property must hold even if the client is paused through expiration, misses a watch event, receives delayed success responses, or resumes with stale local state.
A scheduler leader may enqueue jobs, update checkpoints, or send emails. If two leaders overlap after a GC pause or network partition, duplicated work can be merely expensive or can corrupt state. Fencing each checkpoint write ensures an old leader cannot overwrite a newer leader’s progress.
Object stores, file systems, and search indexes often sit outside the lock service. A worker that holds a lease should include its fencing token with every manifest update or batch commit. The storage layer should reject stale tokens before making the write visible.
etcd’s election API returns a leader key with a creation revision that can be tested in transactions. ZooKeeper exposes globally ordered zxids and per-znode versions. Those monotonic values are useful because they let later leaders prove freshness to systems that need more than a best-effort lock.
A token generator must not hand out the same or lower token after failover. That usually means using a linearizable database sequence, ZooKeeper, etcd, Consul sessions with modify indexes, or a Raft-backed service. Independent Redis nodes with local counters do not provide one global monotonic sequence without another consensus layer.
Store highest_fence_token next to the protected row, object manifest, shard lease, or job checkpoint. Update it in the same transaction as the protected mutation. This makes stale-owner rejection auditable and testable.
UPDATE shard_checkpoint
SET highest_fence_token = :token,
last_processed_offset = :offset,
updated_by = :worker
WHERE shard_id = :shard
AND highest_fence_token <= :token;If duplicate work only wastes money, a best-effort lock with a clear failure mode may be enough. If duplicate work can corrupt data, the protected operation needs fencing, idempotency keys, compare-and-swap, or serializable transactions.