Postgres-Backed Durable Workflow Execution

Distributed Systems advanced Building with AI ~35 min read

Overview

Durable workflow execution solves a narrow but painful systems problem: how to keep multi-step programs moving forward when the process crashes halfway through a payment, data pipeline, webhook handler, or long-running agent task. Instead of rebuilding state by hand after every failure, a durable workflow runtime checkpoints workflow inputs and completed step outputs so recovery can restart from the last safe boundary.

DBOS shows one concrete version of this pattern built directly on PostgreSQL. The application links a library into ordinary Go code, registers workflow functions, and lets the library persist workflow state in Postgres. There is no separate orchestration server on the critical path. The core trade is simple: more database writes in exchange for restart safety, replay control, queueing, and visibility into in-flight work.

This pattern is distinct from a plain job queue. A queue tracks that some work should run; a durable workflow tracks which steps already finished, which one failed, whether a timeout or sleep must survive restart, and whether version skew makes recovery unsafe. That makes it a useful building block for stateful backend automation, exactly-once event handlers, and any workflow that cannot afford to repeat completed side effects blindly.

Core Concepts

Checkpointed Step Boundaries

Workflow code is broken into deterministic orchestration logic plus step functions that do the real work. The runtime persists workflow inputs and the outputs of completed steps. On replay, completed steps return their checkpointed outputs instead of re-executing.

Deterministic Workflow Logic

The workflow function itself must make the same control-flow decisions when rerun with the same inputs and step results. Randomness, current time, direct database reads, and third-party API calls belong in steps, not in the workflow body, because recovery depends on replaying the same step graph.

Application-Library Orchestration

In DBOS, orchestration lives inside the application process rather than in an external workflow server. Postgres stores checkpoints, queue state, and results, while the library handles launch, recovery, and execution. That keeps architecture small for teams already operating Postgres.

Key Concept: Replay Without Repeating Completed Work

A durable workflow runtime does not resume by restoring a frozen stack frame. It reruns the workflow function, intercepts completed steps through stored checkpoints, and continues only when it reaches the first uncheckpointed step.

How It Works

Register Workflows and Run Steps

The DBOS Go API exposes a small surface: create a DBOS context, register a workflow, launch the runtime, and run a workflow. Inside the workflow, ordinary functions become durable step boundaries through RunAsStep.

package main

import (
    "context"
    "fmt"
    "os"
    "time"

    "github.com/dbos-inc/dbos-transact-golang/dbos"
)

func workflow(ctx dbos.DBOSContext, _ string) (string, error) {
    if _, err := dbos.RunAsStep(ctx, stepOne); err != nil {
        return "", err
    }
    return dbos.RunAsStep(ctx, stepTwo)
}

func stepOne(ctx context.Context) (string, error) {
    fmt.Println("step one completed")
    return "step-one", nil
}

func stepTwo(ctx context.Context) (string, error) {
    fmt.Println("step two completed")
    return "step-two", nil
}

func main() {
    dbosCtx, err := dbos.NewDBOSContext(context.Background(), dbos.Config{
        DatabaseURL: os.Getenv("DBOS_SYSTEM_DATABASE_URL"),
        AppName:     "myapp",
    })
    if err != nil {
        panic(err)
    }

    dbos.RegisterWorkflow(dbosCtx, workflow)
    if err := dbos.Launch(dbosCtx); err != nil {
        panic(err)
    }
    defer dbos.Shutdown(dbosCtx, 2*time.Second)

    handle, err := dbos.RunWorkflow(dbosCtx, workflow, "")
    if err != nil {
        panic(err)
    }
    _, err = handle.GetResult()
    if err != nil {
        panic(err)
    }
}

Recover by Replaying Deterministically

When a process dies, DBOS restarts the workflow with its original inputs. Before each step, it checks whether the step output was already checkpointed. If so, recovery returns the stored value immediately; if not, execution resumes normally from that point. This is why deterministic step order matters.

Handle Long Delays and Concurrency Safely

The workflow tutorial adds durable sleep plus deterministic concurrency helpers. Sleep stores its wake-up time in Postgres so a restart does not erase timers. Go and Select let workflows launch concurrent steps while checkpointing the selected branch and result so replay does not depend on scheduler timing.

func raceReplicas(ctx dbos.DBOSContext, key string) (string, error) {
    primaryCh, err := dbos.Go(ctx, func(stepCtx context.Context) (string, error) {
        return queryPrimary(stepCtx, key)
    })
    if err != nil {
        return "", err
    }

    replicaCh, err := dbos.Go(ctx, func(stepCtx context.Context) (string, error) {
        return queryReplica(stepCtx, key)
    })
    if err != nil {
        return "", err
    }

    result, err := dbos.Select(ctx, []<-chan dbos.StepOutcome[string]{primaryCh, replicaCh})
    if err != nil {
        return "", err
    }
    return result, nil
}

Practical Applications

Payments and Business Transactions

Checkout-style workflows often cross several failure domains: charge a card, reserve inventory, create a shipment, send confirmation, update analytics. Durable execution prevents a crash between those stages from forcing operators to guess which actions already happened.

Exactly-Once Event Processing

DBOS uses workflow IDs to turn external events into durable runs. A webhook or Kafka message can start a workflow under a stable identifier so retries from the event source do not create duplicate processing runs.

Long-Running Agents and Data Pipelines

Agents, ingestion pipelines, and maintenance jobs often sleep, poll, or wait on remote systems. Durable workflows turn those waits into stored state instead of fragile in-memory timers, which is especially useful when tasks run for hours or days.

Implementation Patterns

Keep Side Effects Inside Idempotent Steps

Recovery can retry an interrupted step. That means steps should either be naturally idempotent or use explicit external idempotency keys. The workflow tutorial states that steps are tried at least once but never re-executed after completion is checkpointed.

Version Workflow Code for Safe Recovery

Long-running workflows make deployment tricky. DBOS versions workflows so a run started on one application version only recovers on compatible code. Blue-green upgrades let old workers drain old workflows while new workers handle new traffic.

Use Atomic Database Transactions When State Must Match a Checkpoint

DBOS transaction docs describe a special transaction step that commits application data and the DBOS checkpoint atomically. That matters when a workflow step updates business tables and must never acknowledge completion separately from that database write.

// Pseudocode sketch of the pattern:
func updateOrderAndCheckpoint(tx Tx, orderID string) error {
    if _, err := tx.Exec(
        "UPDATE orders SET status = 'paid' WHERE id = $1",
        orderID,
    ); err != nil {
        return err
    }

    // DBOS commits the workflow checkpoint in the same transaction.
    return nil
}

Performance Considerations

Common Pitfalls

Further Reading