Serverless Postgres Compute-Storage Separation

Database Engineering advanced Building with AI ~35 min read

Overview

Traditional PostgreSQL deployments bind database compute, buffer cache, WAL, and persistent storage to the same host or volume. That design is simple and fast for local disks, but it makes cloud operations rigid: storage must be provisioned ahead of time, compute cannot scale to zero without taking the database with it, and realistic clones require slow backup or snapshot flows. Serverless Postgres architectures break that coupling by treating the Postgres process as an elastic compute layer and moving durable state into a distributed storage service.

Databricks describes Lakebase as a Postgres-compatible operational database that places transactional data on lake-style cloud storage while allowing compute to run independently. The underlying pattern is the same one public Neon architecture material explains in more detail: compute nodes run PostgreSQL, write-ahead log is streamed to durable safekeepers, pageservers materialize page versions for reads, and object storage becomes the long-term source of truth.

The transferable lesson is not a vendor feature checklist. It is a storage-engine boundary: keep SQL semantics and extension compatibility in ordinary Postgres compute, but intercept page persistence below the engine so durability, branching, point-in-time recovery, archive storage, and scale-to-zero can be managed by a cloud-native storage layer.

Core Concepts

Stateless Postgres Compute

A compute node runs the PostgreSQL process and accepts client connections, but it is not the only durable owner of database pages. It can be paused, restarted, resized, or replaced because persistent page history lives in the storage service. The compute still owns transaction execution, query planning, buffer management, locks, and SQL-visible behavior while delegating long-term page durability.

WAL as the Storage Contract

PostgreSQL already expresses durable changes through write-ahead log. A separated architecture uses WAL as the handoff between compute and storage: the compute streams WAL records to a replicated write buffer, and downstream storage components reorganize those records into page versions that can be fetched by relation block and log sequence number.

Pageservers and Object Storage

Pageservers act as the low-latency read path and cache. They ingest WAL, maintain a storage format optimized for page reconstruction, serve page requests from compute nodes, and upload immutable layer files to cloud object storage. Object storage supplies cheap durable capacity, while pageservers keep hot data close enough for database workloads.

Key Concept: Separate Durability from Execution

Serverless Postgres does not make SQL execution stateless. It makes the compute process replaceable by moving the durable page timeline into a storage service that can outlive any single Postgres instance.

How It Works

Write Path

On writes, the compute node follows PostgreSQL transaction rules and emits WAL. Instead of relying only on local disk, it streams the WAL to safekeepers. Safekeepers replicate recent WAL, often across availability zones, so the system can acknowledge durable progress before slower background processing has compacted data into long-term object storage.

# Simplified write path for separated Postgres storage.
txn = postgres.execute(sql)
wal_records = txn.emit_wal()

quorum = safekeepers.append(
    tenant_id=tenant,
    timeline_id=timeline,
    start_lsn=txn.start_lsn,
    records=wal_records,
)

if quorum.durable_lsn >= txn.commit_lsn:
    client.acknowledge_commit()

pageserver.ingest_async(wal_records)
object_store.upload_layers_async(timeline)

Read Path

On reads, the compute asks the storage layer for a page at a specific log sequence number. The pageserver locates the newest base image for the relation block, applies any newer WAL deltas needed to reach the requested LSN, and returns the reconstructed page. Hot pages can be served from memory or SSD cache; cold pages may require fetching layer files from object storage.

def get_page_at_lsn(tenant, timeline, rel_block, lsn):
    image = layer_index.find_base_image(rel_block, at_or_before=lsn)
    deltas = layer_index.find_wal_deltas(rel_block, after=image.lsn, to=lsn)
    page = image.bytes
    for delta in deltas:
        page = postgres_redo(page, delta)
    return page

Branching and Restore

Because the storage layer stores page history by timeline and LSN, branches are metadata operations over a shared page history rather than full physical copies. A child timeline points to its parent at a branch LSN and writes new page versions only after divergence. Point-in-time restore is similar: start compute against an earlier LSN and let storage reconstruct pages from the retained history.

Practical Applications

Scale-to-Zero Development Databases

When compute is replaceable, inactive databases can stop paying for running Postgres processes. The durable timeline remains in storage; a new compute process attaches when traffic resumes. This is useful for preview environments, per-developer databases, and applications with bursty usage where idle compute would dominate cost.

Production-Like Branches for Migration Safety

Database branches let teams run schema migrations, index experiments, and data repairs against a realistic copy without duplicating the whole database. The branch shares historical pages until it writes new ones, which makes isolated experiments practical for CI jobs and automated coding agents that need safe database credentials.

Transactional and Analytical Integration

Lakebase-style systems emphasize putting transactional data closer to lakehouse storage and governance. The engineering goal is to reduce export pipelines between OLTP and analytics while keeping Postgres semantics for application writes. The hard part is preserving low-latency transaction behavior while exposing data to broader analytical workflows.

Implementation Patterns

Timeline-Oriented Storage Metadata

Model storage around tenants, timelines, relation blocks, and LSN ranges. This makes branching, restore, retention, and garbage collection explicit. The pageserver can then answer “which layer contains this block at this LSN?” without scanning object storage directly.

CREATE TABLE page_layers (
    tenant_id     text NOT NULL,
    timeline_id   text NOT NULL,
    relation_id   text NOT NULL,
    block_start   bigint NOT NULL,
    block_end     bigint NOT NULL,
    lsn_start     pg_lsn NOT NULL,
    lsn_end       pg_lsn NOT NULL,
    object_key    text NOT NULL,
    layer_kind    text NOT NULL CHECK (layer_kind IN ('image', 'delta')),
    PRIMARY KEY (tenant_id, timeline_id, relation_id, block_start, lsn_start)
);

Separate Hot, Recent, and Cold State

Do not send every page request to object storage. Recent WAL belongs in a replicated write buffer, hot page layers belong in memory or SSD near pageservers, and cold immutable layers can live in object storage. The value of the architecture depends on this tiering: object storage is durable and cheap, but the database read path needs a cache hierarchy.

Compatibility Through a Narrow Storage Shim

The less PostgreSQL is modified, the easier it is to track upstream releases and support extensions. Production systems typically keep SQL behavior in Postgres and build a narrow shim around page reads, WAL streaming, startup control, and connection proxying. That boundary should be tested like a storage engine: crash recovery, torn writes, stale caches, and timeline divergence all need explicit coverage.

Performance Considerations

Common Pitfalls

Further Reading