In a multi-group Raft architecture, a single physical node may host dozens or hundreds of independent Raft groups (one per shard or partition). Each group needs to exchange AppendEntries, RequestVote, and InstallSnapshot messages with its peers on other nodes. A naive approach — opening a dedicated connection per Raft group — quickly exhausts file descriptors and creates head-of-line blocking. Multi-Raft HTTP transport solves this by multiplexing all Raft group traffic over a shared set of connections between each pair of nodes.
A distributed database system with many shards may implement a custom HTTP-based transport layer for exactly this purpose. Each inter-node HTTP connection carries messages for many Raft groups, tagged by group ID, and the transport layer demultiplexes inbound messages to the correct Raft state machine on the receiving end.
Choosing HTTP as the wire protocol (rather than raw TCP or gRPC) brings operational advantages: standard load balancers and proxies can inspect and route traffic, TLS termination uses existing PKI infrastructure, and debugging is easier with standard tooling. The trade-off is slightly higher per-message overhead, which is acceptable for most Raft workloads where messages are already batched.
Multi-Raft transport separates the logical topology (which Raft groups talk to which) from the physical topology (which nodes connect to which). This separation lets the system scale to thousands of Raft groups without proportional connection growth.
HTTP/2 multiplexing is a natural fit for multi-Raft transport: each Raft group can use its own HTTP/2 stream within a single TCP connection, getting independent flow control without head-of-line blocking at the application layer.
Each outbound Raft message is wrapped in an envelope that includes the source node ID, destination node ID, Raft group ID, and the serialized Raft message (protobuf or custom binary). The envelope is sent as the body of an HTTP POST to a well-known path on the destination node (e.g., /raft/message).
The transport maintains a connection pool per remote peer. Connections are created lazily on first message to a peer and kept alive with periodic health checks. When a peer becomes unreachable, the transport marks it as down and buffers messages (up to a limit) for retry. Messages to different Raft groups on the same peer share the same connection pool.
The receiving node runs an HTTP server that accepts Raft messages on a dedicated endpoint. It extracts the group ID from the envelope, looks up the corresponding Raft node in a registry, and delivers the message. Unknown group IDs are logged and dropped (they may indicate a stale configuration).
Snapshot transfers require special handling: they are large (megabytes to gigabytes) and must not block regular Raft heartbeats. The transport uses a separate endpoint (e.g., /raft/snapshot) with chunked transfer encoding and a dedicated connection to avoid starving time-sensitive messages.
// Go: Multi-Raft HTTP transport handler
type RaftTransport struct {
mu sync.RWMutex
groups map[uint64]*raft.RawNode // groupID -> Raft node
pools map[string]*http.Client // peerAddr -> connection pool
snapCh chan SnapshotRequest
}
func (t *RaftTransport) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/raft/message":
t.handleMessage(w, r)
case "/raft/snapshot":
t.handleSnapshot(w, r)
default:
http.NotFound(w, r)
}
}
func (t *RaftTransport) handleMessage(w http.ResponseWriter, r *http.Request) {
var env RaftEnvelope
if err := json.NewDecoder(r.Body).Decode(&env); err != nil {
http.Error(w, "bad request", http.StatusBadRequest)
return
}
t.mu.RLock()
node, ok := t.groups[env.GroupID]
t.mu.RUnlock()
if !ok {
http.Error(w, "unknown group", http.StatusNotFound)
return
}
if err := node.Step(env.Message); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
}// Go: Sending messages with connection pooling
func (t *RaftTransport) Send(peerAddr string, env RaftEnvelope) error {
client := t.getOrCreatePool(peerAddr)
body, err := json.Marshal(env)
if err != nil {
return fmt.Errorf("marshal envelope: %w", err)
}
resp, err := client.Post(
fmt.Sprintf("http://%s/raft/message", peerAddr),
"application/json",
bytes.NewReader(body),
)
if err != nil {
return fmt.Errorf("send to %s: %w", peerAddr, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("peer %s returned %d", peerAddr, resp.StatusCode)
}
return nil
}
func (t *RaftTransport) getOrCreatePool(addr string) *http.Client {
t.mu.RLock()
if c, ok := t.pools[addr]; ok {
t.mu.RUnlock()
return c
}
t.mu.RUnlock()
t.mu.Lock()
defer t.mu.Unlock()
if c, ok := t.pools[addr]; ok {
return c
}
c := &http.Client{
Transport: &http.Transport{
MaxIdleConnsPerHost: 4,
IdleConnTimeout: 90 * time.Second,
},
Timeout: 10 * time.Second,
}
t.pools[addr] = c
return c
}The multi-Raft HTTP transport pattern appears in several production distributed databases: