Append-Only Cloud Database Storage Nodes

Database Engineering advanced Building with AI ~35 min read

Overview

Cloud databases often separate compute from storage so read replicas, scale-out capacity, and failure recovery do not depend on copying whole database volumes. Taurus, a cloud database architecture published at SIGMOD, follows that broad pattern but makes a more specific storage-engine choice: the hot storage path is append-only, log records are synchronously replicated into bounded persistent logs, and page stores materialize database pages from those logs.

The design targets three properties that are usually in tension. It wants write latency close to a local database, availability under storage-node failures, and lower hardware cost than every compute replica maintaining its own complete durable copy. Taurus gets there by splitting responsibilities between database compute, Log Stores, and Page Stores while keeping critical write paths to at most one network hop.

The transferable lesson is that a disaggregated database storage layer should not just be a remote disk. It can expose log-oriented primitives that match database recovery semantics, use append-only writes to reduce random I/O and device wear, and let storage nodes perform redo, snapshots, and replica catch-up without forcing the primary compute node to ship pages around.

Core Concepts

Compute and Storage Disaggregation

The database master executes transactions, maintains the buffer pool, and emits redo records. Durable state is pushed into a storage service rather than being owned exclusively by the compute host. Read replicas can fetch log or page state from storage nodes, which reduces the amount of replication traffic that must flow through the master.

Persistent Logs

A Persistent Log, or PLog, is a bounded append-only object replicated across Log Store nodes. The database writes redo into PLogs; once the configured replicas have accepted the append, the log record is durable. Bounded logs make allocation, sealing, recovery, and garbage collection tractable because each log object has a finite lifetime and a known replica set.

Page Stores

Page Stores own slices of the database page space. They consume redo records, apply them to page images, and serve page reads to compute nodes and replicas. This lets page reconstruction move out of the primary database process, so recovery and replica reads do not require the master to replay every change for every consumer.

Key Concept: Log-First Storage

The durable contract is an append to a replicated log, not a synchronous overwrite of a remote page. Page images are derived artifacts that can be rebuilt, cached, snapshotted, and compacted by storage nodes.

How It Works

Write Path

When the database commits a transaction, it emits redo records and appends them to the current PLog. The Log Store replica group performs synchronous replication for that append. The compute node can acknowledge commit once the log durability rule is satisfied, while Page Stores asynchronously consume the same log stream to update page state.

def commit_transaction(txn):
    redo = txn.redo_records()
    plog = log_allocator.current_plog(txn.database_id)

    result = log_store.append(
        plog_id=plog.id,
        lsn=txn.commit_lsn,
        records=redo,
        replication_factor=3,
    )

    if result.durable:
        client.acknowledge_commit()
        page_store_queue.publish(redo)

Read and Recovery Path

For reads that miss the compute buffer pool, the compute node asks the Page Store responsible for that page slice. The Page Store can return a current page image or apply pending redo to bring a page forward. During recovery, a replacement compute node can rebuild state by attaching to the storage service rather than scanning a full local volume.

Snapshots

Append-only storage makes snapshots cheap because older log and page versions do not need to be rewritten in place. A snapshot can record stable log positions and page-slice metadata, then rely on immutable or append-only objects to preserve the referenced state. The system still needs retention and garbage collection, but snapshot creation is not proportional to database size.

Practical Applications

Cloud OLTP Databases

The pattern is useful for managed relational databases that need high availability without making every replica perform the full write path. A primary compute node can keep transaction execution fast while storage nodes handle durable replication, page serving, and recovery responsibilities.

Read Replica Fan-Out

Read replicas can catch up from shared storage logs and page stores instead of receiving every page or log record directly from the master. That reduces master network pressure and makes replica creation closer to a metadata operation than a full data copy.

Fast Failover

If compute is replaceable and durable redo already lives in replicated Log Stores, failover focuses on choosing a new compute owner and replaying the necessary log tail. The storage system becomes the stable substrate that survives compute restarts and host failures.

Implementation Patterns

Bounded Log Segments

Use finite log objects rather than one unbounded stream. Each segment can be allocated to a replica group, sealed when full, indexed by LSN range, and garbage-collected when no page store, replica, or snapshot needs it.

CREATE TABLE plog_segments (
    plog_id        text PRIMARY KEY,
    database_id    text NOT NULL,
    lsn_start      bigint NOT NULL,
    lsn_end        bigint,
    state          text NOT NULL CHECK (state IN ('open', 'sealed', 'gc_ready')),
    replica_nodes  text[] NOT NULL
);

Page-Slice Ownership

Shard page ownership independently from compute placement. A database can have many page slices, and each Page Store is responsible for a subset. That makes recovery and read serving parallelizable, but it requires a reliable metadata service that maps page identifiers to current slice owners.

Redo-Aware Storage APIs

Expose APIs in terms of log sequence numbers, page identifiers, and durability acknowledgements. A generic object-store API forces the database to rebuild its own recovery protocol above storage; a database-aware API lets the storage service provide exactly the invariants the engine needs.

Performance Considerations

Common Pitfalls

Further Reading