Every managed cloud service—database-as-a-service, compute instances, managed Kafka clusters—faces the same design challenge: instances have complex lifecycles that span multiple asynchronous operations, and the system must track, enforce, and communicate these states reliably. A database instance isn’t simply “on” or “off”; it moves through states like pending, provisioning, ready, updating, degraded, suspended, deprovisioning, and deleted, each with its own set of allowed transitions and operational semantics.
The state machine pattern formalizes these transitions into an explicit, auditable model. Each state defines what operations are allowed (you can’t delete an instance that’s mid-provisioning), what the user sees (a “provisioning” instance shows a spinner, not a connection string), and what the system is doing (a background job is allocating storage and configuring replication). The transition function validates that every state change is legal, preventing the system from entering impossible states like “ready but no storage allocated.”
As a practical example, a managed database service can implement instance lifecycle management with a PostgreSQL-backed state machine, async provisioning via a job queue, and a reconciliation loop that detects and recovers from stuck transitions.
An instance lifecycle is modeled as a finite state machine: a set of states, a set of valid transitions between states, and a transition function that determines whether a given state change is allowed. The FSM is the single source of truth for “what can happen next.” All state changes must go through the FSM—no direct database UPDATEs to the state column. This ensures that invariants (like “you must provision before you can be ready”) are always maintained.
Provisioning a database instance involves slow, fallible operations: allocating disk, creating replication slots, configuring networking, initializing schemas. These cannot happen synchronously in an API request. Instead, the API creates the instance record in “pending” state and enqueues a provisioning job. A background worker picks up the job, transitions the instance to “provisioning,” performs the work, and transitions to “ready” on success or “failed” on error. The job queue provides at-least-once delivery, retry with backoff, and dead-letter handling.
Every state transition is recorded in an append-only audit log with the previous state, new state, timestamp, actor (user or system), and reason. This log is essential for debugging (“why is this instance stuck in provisioning?”), compliance (“when was this instance deleted and by whom?”), and SLA measurement (“how long does provisioning typically take?”). The audit log is never modified—it is an immutable event stream.
Distributed systems fail in unexpected ways: a provisioning worker crashes mid-operation, a network partition causes a state update to be lost, or an external dependency (cloud provider API) returns an error after partially completing. The reconciliation loop runs periodically, compares the declared state (what the database says) against the actual state (what the infrastructure shows), and triggers corrective actions. Instances stuck in transitional states beyond a timeout are either retried or marked as failed.
When a worker transitions an instance from “provisioning” to “ready,” another process might concurrently attempt to transition it to “suspended” (due to a billing event). Optimistic locking prevents lost updates: the UPDATE includes a WHERE state = 'provisioning' clause, and the worker checks that exactly one row was affected. If zero rows were updated, the state changed out from under the worker, and it must re-read and re-evaluate.
The implementation has three components: the state machine definition, the PostgreSQL-backed persistence layer with optimistic locking, and the async job processor.
package lifecycle
type InstanceState string
const (
StatePending InstanceState = "pending"
StateProvisioning InstanceState = "provisioning"
StateReady InstanceState = "ready"
StateUpdating InstanceState = "updating"
StateDegraded InstanceState = "degraded"
StateSuspended InstanceState = "suspended"
StateDeprovisioning InstanceState = "deprovisioning"
StateDeleted InstanceState = "deleted"
StateFailed InstanceState = "failed"
)
var validTransitions = map[InstanceState][]InstanceState{
StatePending: {StateProvisioning, StateFailed},
StateProvisioning: {StateReady, StateFailed},
StateReady: {StateUpdating, StateDegraded, StateSuspended, StateDeprovisioning},
StateUpdating: {StateReady, StateDegraded, StateFailed},
StateDegraded: {StateReady, StateDeprovisioning, StateFailed},
StateSuspended: {StateReady, StateDeprovisioning},
StateDeprovisioning: {StateDeleted, StateFailed},
// StateFailed and StateDeleted are terminal — no transitions out
}
func CanTransition(from, to InstanceState) bool {
allowed, ok := validTransitions[from]
if !ok {
return false
}
for _, s := range allowed {
if s == to {
return true
}
}
return false
}package lifecycle
import (
"context"
"database/sql"
"fmt"
"time"
)
type Instance struct {
ID string
OrgID string
State InstanceState
Region string
Plan string
StateAt time.Time
CreatedAt time.Time
}
func TransitionState(ctx context.Context, db *sql.DB,
instanceID string, from, to InstanceState, reason string) error {
if !CanTransition(from, to) {
return fmt.Errorf("invalid transition: %s -> %s", from, to)
}
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
// Optimistic lock: only update if current state matches expected
res, err := tx.ExecContext(ctx,
`UPDATE instances
SET state = $1, state_changed_at = now(), updated_at = now()
WHERE id = $2 AND state = $3`,
to, instanceID, from)
if err != nil {
return fmt.Errorf("update state: %w", err)
}
rows, _ := res.RowsAffected()
if rows == 0 {
return fmt.Errorf("state conflict: instance %s is no longer in state %s",
instanceID, from)
}
// Append to audit log
_, err = tx.ExecContext(ctx,
`INSERT INTO instance_state_log
(instance_id, from_state, to_state, reason, created_at)
VALUES ($1, $2, $3, $4, now())`,
instanceID, from, to, reason)
if err != nil {
return fmt.Errorf("audit log: %w", err)
}
return tx.Commit()
}package lifecycle
import (
"context"
"database/sql"
"log"
"time"
)
type ProvisioningWorker struct {
DB *sql.DB
Infra InfraProvisioner
PollRate time.Duration
}
type InfraProvisioner interface {
AllocateStorage(ctx context.Context, instanceID, region string) error
ConfigureNetwork(ctx context.Context, instanceID, region string) error
InitializeDatabase(ctx context.Context, instanceID string) error
}
func (w *ProvisioningWorker) Run(ctx context.Context) {
ticker := time.NewTicker(w.PollRate)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
w.processPendingInstances(ctx)
}
}
}
func (w *ProvisioningWorker) processPendingInstances(ctx context.Context) {
rows, err := w.DB.QueryContext(ctx,
`SELECT id, region FROM instances
WHERE state = 'pending'
ORDER BY created_at
LIMIT 10
FOR UPDATE SKIP LOCKED`)
if err != nil {
log.Printf("query pending instances: %v", err)
return
}
defer rows.Close()
for rows.Next() {
var id, region string
if err := rows.Scan(&id, ®ion); err != nil {
continue
}
go w.provision(ctx, id, region)
}
}
func (w *ProvisioningWorker) provision(ctx context.Context, id, region string) {
err := TransitionState(ctx, w.DB, id,
StatePending, StateProvisioning, "provisioning started")
if err != nil {
log.Printf("transition to provisioning: %v", err)
return
}
steps := []struct {
name string
fn func(context.Context, string, string) error
}{
{"allocate_storage", w.Infra.AllocateStorage},
{"configure_network", w.Infra.ConfigureNetwork},
}
for _, step := range steps {
if err := step.fn(ctx, id, region); err != nil {
TransitionState(ctx, w.DB, id,
StateProvisioning, StateFailed,
fmt.Sprintf("%s failed: %v", step.name, err))
return
}
}
if err := w.Infra.InitializeDatabase(ctx, id); err != nil {
TransitionState(ctx, w.DB, id,
StateProvisioning, StateFailed,
fmt.Sprintf("init_db failed: %v", err))
return
}
TransitionState(ctx, w.DB, id,
StateProvisioning, StateReady, "provisioning complete")
}The instance lifecycle state machine is the control plane backbone for a managed database service. When a customer clicks “Create Database” in the dashboard, the API creates an instance in pending state and returns immediately with a 202 Accepted. The dashboard polls the instance status and shows a progress indicator as it moves through provisioning to ready. If provisioning fails, the customer sees a clear error state with the option to retry.
The suspension flow handles billing failures gracefully. When Stripe reports a failed payment (via webhook), the billing service transitions the instance to suspended. In this state, the instance is still running but rejects write operations—reads continue so the customer doesn’t lose access to their data. When payment is resolved, the instance transitions back to ready. This design balances revenue protection with customer experience.
The reconciliation loop catches infrastructure drift. If a Kubernetes pod backing an instance crashes and restarts, the health checker detects the degradation and transitions the instance to degraded. The reconciliation loop then attempts self-healing (restarting the pod, reprovisioning storage) and transitions back to ready if successful. Operators see the full state history in the audit log and can intervene if automated recovery fails.
Transitional states (provisioning, updating, deprovisioning) should have maximum durations. A reconciliation query like SELECT * FROM instances WHERE state = 'provisioning' AND state_changed_at < now() - INTERVAL '30 minutes' finds stuck instances. The reconciler can retry the operation, escalate to an operator, or transition to failed with a timeout reason.
Because provisioning can be retried after failure, each step must be idempotent. “Allocate storage” should check if storage already exists before creating it. “Configure network” should be a declarative apply, not an imperative create. This allows the system to restart provisioning from the beginning without creating duplicate resources or encountering “already exists” errors.
State transitions emit events (via PostgreSQL NOTIFY or a message queue) that downstream systems consume. The notification service sends emails when provisioning completes. The metrics service records transition latencies. The billing service activates metering when an instance reaches ready. This decoupling keeps the state machine simple—it only manages transitions, not side effects.
state = 'ready' directly skips validation, audit logging, and event emission. All state changes must go through the transition function.WHERE state = expected clause, concurrent transitions can overwrite each other. A billing suspension and a provisioning completion can race, leaving the instance in an inconsistent state.deleted or failed states breaks the invariant that these are final. If recovery is needed, create a new instance; don’t resurrect a deleted one.