Hybrid Logical Clocks (HLC) solve a fundamental tension in distributed systems: physical wall-clock time is useful for humans and for bounding staleness, but it cannot guarantee causal ordering because clocks drift. Pure logical clocks (Lamport clocks) guarantee causality but produce timestamps that bear no relation to real time. HLC combines both — each timestamp has a physical component bounded close to wall-clock time and a logical counter that breaks ties, giving you causally consistent ordering with timestamps that are meaningful to humans.
HLC was introduced by Kulkarni et al. (2014) and has been adopted by systems like CockroachDB, YugabyteDB, and similar distributed databases. In these systems, every transaction and event receives an HLC timestamp that serves as its MVCC version. Because HLC timestamps respect causality, the system can determine “which write happened first” without requiring synchronized physical clocks, while still supporting time-bounded queries like “read as of 5 seconds ago.”
For a distributed database’s multi-node architecture, HLC is the backbone of transaction ordering, snapshot isolation, and cross-shard consistency. Understanding how HLC updates on local events, sends, and receives is essential for reasoning about consistency guarantees and debugging timestamp anomalies.
a happens before event b (a → b), then hlc(a) < hlc(b). The converse is not guaranteed — HLC captures causal order, not equivalence.HLC timestamps are tuples (physical_time, logical_counter, node_id). The physical component stays close to real time, the logical counter ensures uniqueness, and the node_id breaks remaining ties. Together they form a total order that respects causality.
Unlike Lamport clocks, HLC’s physical component gives timestamps real-world meaning. You can use an HLC timestamp to answer “approximately when did this event happen?” — the answer is always within the cluster’s maximum clock skew of the truth.
HLC maintains two variables per node: pt (physical) and l (logical). The algorithm has three cases:
pt = max(pt, now()). If pt didn’t change (physical clock hasn’t advanced), increment l. Otherwise reset l = 0.pt = max(pt, msg.pt, now()). If the new pt equals the old pt and msg.pt, set l = max(l, msg.l) + 1. If it equals only one of them, set l = max of the matching counter + 1. If now() is the winner, reset l = 0.(pt, l) as the event’s HLC timestamp.The logical counter only increments when the physical clock is stuck. In practice, with millisecond-resolution clocks and events spaced even microseconds apart, the logical counter stays small. This keeps timestamps compact and human-readable.
In distributed databases like CockroachDB and similar systems, every write gets an HLC timestamp as its MVCC version. Reads at a specific timestamp see all writes with HLC ≤ that timestamp. Because HLC respects causality, this provides snapshot isolation: a transaction that reads its own writes will always see them, and concurrent transactions get consistent snapshots.
When a node receives a message with a physical component far ahead of its local clock, the HLC algorithm still works correctly — but the node’s timestamps jump forward. Systems monitor the gap between pt and now() to detect excessive skew. If the gap exceeds a configured threshold (e.g., 500ms), the node can refuse transactions or alert operators, because large skew means uncertainty windows widen and time-bounded queries become less precise.
pt and now() on every node; alert when it exceeds half your maximum tolerable skew.[commit_ts, commit_ts + max_offset] to handle reads that might observe writes from nodes with faster clocks.type HLC struct {
mu sync.Mutex
pt int64 // physical time (ms since epoch)
l int32 // logical counter
nodeID uint16
wallClock func() int64
}
func (h *HLC) Now() Timestamp {
h.mu.Lock()
defer h.mu.Unlock()
now := h.wallClock()
if now > h.pt {
h.pt = now
h.l = 0
} else {
h.l++
}
return Timestamp{Physical: h.pt, Logical: h.l, NodeID: h.nodeID}
}
func (h *HLC) Update(msg Timestamp) Timestamp {
h.mu.Lock()
defer h.mu.Unlock()
now := h.wallClock()
oldPt := h.pt
if now > oldPt && now > msg.Physical {
h.pt = now
h.l = 0
} else if oldPt == msg.Physical {
h.pt = oldPt
h.l = max(h.l, msg.Logical) + 1
} else if oldPt > msg.Physical {
h.pt = oldPt
h.l++
} else {
h.pt = msg.Physical
h.l = msg.Logical + 1
}
return Timestamp{Physical: h.pt, Logical: h.l, NodeID: h.nodeID}
}type Timestamp struct {
Physical int64
Logical int32
NodeID uint16
}
func (t Timestamp) Less(other Timestamp) bool {
if t.Physical != other.Physical {
return t.Physical < other.Physical
}
if t.Logical != other.Logical {
return t.Logical < other.Logical
}
return t.NodeID < other.NodeID
}
// Pack into a single uint64 for storage (48-bit physical + 16-bit logical)
func (t Timestamp) Pack() uint64 {
return uint64(t.Physical)<<16 | uint64(t.Logical)&0xFFFF
}
func UnpackTimestamp(v uint64) Timestamp {
return Timestamp{
Physical: int64(v >> 16),
Logical: int32(v & 0xFFFF),
}
}max_offset.now() jump backward, causing pt to stall until wall-clock catches up. Use monotonic clock sources or NTP slew mode instead of step adjustments.