Many backend operations are too slow or too unreliable to handle synchronously within an HTTP request: sending emails, provisioning infrastructure, processing uploads, running backups. These operations need to be enqueued and processed asynchronously, with guarantees around delivery, retry, and ordering. While dedicated message brokers like RabbitMQ or cloud services like SQS are common choices, there’s a compelling alternative: using the PostgreSQL database you already run as your job queue.
Gue is a Go library that implements a persistent job queue backed by PostgreSQL. It uses PostgreSQL’s LISTEN/NOTIFY for near-instant job delivery, advisory locks for safe concurrent processing, and transactions to ensure that enqueueing a job and updating application state happen atomically—if the transaction rolls back, the job is never enqueued. This transactional guarantee eliminates an entire class of consistency bugs where the application state changes but the async job is lost (or vice versa).
In a typical SaaS backend, Gue powers async workflows for instance provisioning, scheduled backups, and long-running data migrations. When a user requests a new database instance, the API handler enqueues a provisioning job in the same transaction that creates the instance record. If the transaction commits, the job is guaranteed to execute; if it fails, neither the instance record nor the job exists. This transactional coupling is the primary advantage of a database-backed queue over an external broker.
The defining feature of a PostgreSQL-backed queue is that enqueuing a job happens inside the same database transaction as your application writes. If you insert a user record and enqueue a “send welcome email” job in the same transaction, either both succeed or neither does. With an external broker, you’d need to implement the outbox pattern or accept eventual consistency between your database and queue.
PostgreSQL’s LISTEN/NOTIFY mechanism allows Gue workers to be notified immediately when a new job is inserted, rather than polling the jobs table on a fixed interval. This reduces job pickup latency from seconds (polling interval) to milliseconds. Workers LISTEN on a channel, and Gue’s enqueue function sends a NOTIFY after the inserting transaction commits.
When multiple worker processes compete for the same job, PostgreSQL advisory locks prevent double-processing. A worker acquires an advisory lock on the job’s ID before processing it; if another worker already holds the lock, the attempt fails immediately without blocking. This is lighter than SELECT ... FOR UPDATE SKIP LOCKED and avoids holding row-level locks for the duration of job processing.
When a job fails (the handler returns an error), Gue re-enqueues it with a delay that increases exponentially: 1 second, then 4, 16, 64, and so on, up to a configurable maximum. Each retry increments an error count on the job row. After a configurable number of retries, the job is marked as permanently failed and moved to a dead-letter state for manual investigation.
Each job has a string type (e.g., "provision_instance", "run_backup", "send_notification") that determines which handler processes it. Workers register handlers for specific types, allowing you to run specialized worker pools: a pool of provisioning workers with elevated cloud permissions, a separate pool of notification workers with SMTP credentials, and so on.
Gue requires a gue_jobs table in your PostgreSQL database. The migration creates the table with columns for job type, payload (JSONB), run-at time, error count, and last error message. An index on (run_at, job_type) ensures efficient polling for ready jobs.
-- Migration: create the gue_jobs table
CREATE TABLE gue_jobs (
job_id BIGSERIAL PRIMARY KEY,
queue TEXT NOT NULL DEFAULT '',
job_type TEXT NOT NULL,
args JSONB NOT NULL DEFAULT '{}',
run_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
error_count INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_gue_jobs_selector
ON gue_jobs (queue, run_at, job_type)
WHERE error_count < 10;The key pattern: open a transaction, perform your application writes, enqueue the job, and commit. Gue’s Enqueue function accepts a transaction handle so the job insert participates in the same transaction.
package provisioning
import (
"context"
"encoding/json"
"github.com/vgarvardt/gue/v5"
"github.com/vgarvardt/gue/v5/adapter/pgxv5"
"github.com/jackc/pgx/v5/pgxpool"
)
type Service struct {
pool *pgxpool.Pool
client *gue.Client
}
type ProvisionArgs struct {
InstanceID string `json:"instance_id"`
Plan string `json:"plan"`
Region string `json:"region"`
}
func (s *Service) CreateInstance(ctx context.Context, args ProvisionArgs) error {
tx, err := s.pool.Begin(ctx)
if err != nil {
return err
}
defer tx.Rollback(ctx)
// Insert the instance record
_, err = tx.Exec(ctx,
`INSERT INTO instances (id, plan, region, status)
VALUES ($1, $2, $3, 'provisioning')`,
args.InstanceID, args.Plan, args.Region,
)
if err != nil {
return err
}
// Enqueue the provisioning job in the same transaction
payload, _ := json.Marshal(args)
j := &gue.Job{
Type: "provision_instance",
Args: payload,
}
if err := s.client.EnqueueTx(ctx, j, pgxv5.NewTx(tx)); err != nil {
return err
}
return tx.Commit(ctx)
}Workers register handler functions for each job type, then start polling. When a job is picked up, the handler receives the parsed arguments and a context that is cancelled if the worker shuts down. Returning an error triggers a retry; returning nil marks the job as complete and deletes it from the table.
package worker
import (
"context"
"encoding/json"
"log"
"time"
"github.com/vgarvardt/gue/v5"
)
func StartWorkers(ctx context.Context, client *gue.Client) error {
wm := gue.WorkMap{
"provision_instance": handleProvision,
"run_backup": handleBackup,
"send_notification": handleNotification,
}
workers, err := gue.NewWorkerPool(client, wm, 4,
gue.WithPoolQueue("default"),
gue.WithPoolPollInterval(5*time.Second),
)
if err != nil {
return err
}
return workers.Run(ctx)
}
func handleProvision(ctx context.Context, j *gue.Job) error {
var args ProvisionArgs
if err := json.Unmarshal(j.Args, &args); err != nil {
return err
}
log.Printf("provisioning instance %s in %s", args.InstanceID, args.Region)
// Call cloud provider APIs to provision the instance...
if err := provisionCloudResources(ctx, args); err != nil {
return err // will be retried with exponential backoff
}
// Update the instance status to "ready"
_, err := db.Exec(ctx,
`UPDATE instances SET status = 'ready', updated_at = NOW()
WHERE id = $1`, args.InstanceID)
return err
}When the worker process receives a shutdown signal (SIGTERM, SIGINT), it should stop accepting new jobs and wait for in-flight jobs to complete. Gue supports this through context cancellation: when the context passed to Run is cancelled, the worker pool finishes processing current jobs and returns. Setting a deadline on the shutdown context ensures the process doesn’t hang indefinitely on a stuck job.
Jobs can be scheduled for future execution by setting the RunAt field. This is useful for delayed operations like “send a reminder email in 24 hours” or “retry provisioning after a 5-minute cooldown.” Workers only pick up jobs whose run_at timestamp is in the past, so scheduled jobs wait in the table until their time arrives.
Because jobs can be retried after partial completion, every handler must be idempotent. For provisioning, this means checking whether the cloud resource already exists before creating it. For notifications, it means recording that a notification was sent and skipping if a record already exists. Design handlers so that running them twice produces the same result as running them once.
Gue supports named queues. By assigning critical jobs (instance provisioning) to a “high” queue and background jobs (analytics aggregation) to a “low” queue, you can run separate worker pools with different concurrency levels. The high-priority pool might run 8 workers while the low-priority pool runs 2, ensuring critical operations get resources even when the background queue is backlogged.
Query the gue_jobs table to monitor queue health: count of pending jobs by type, jobs with high error counts (approaching the dead-letter threshold), and the oldest un-processed job (indicating a stuck queue). Export these metrics to your monitoring system and alert when the queue depth or age exceeds thresholds.
LISTEN/NOTIFY, workers that poll every 100ms create unnecessary database load. Use notification-based delivery for low latency, and set the poll interval to 5–30 seconds as a fallback for missed notifications.