Cron-Based Job Scheduling Systems

Distributed Systems intermediate Building with AI 14 min read

Overview

Cron-based job scheduling is the backbone of periodic task execution in server-side applications. From generating nightly reports to syncing data with external APIs every five minutes, cron schedules define when work happens. While Unix cron is the classic implementation, modern applications embed scheduling directly into their process using libraries like robfig/cron (Go) that support second-level precision, dynamic job management, and programmatic control.

In a typical server-side application, a cron scheduler drives periodic health checks, data aggregation, cache warming, and cleanup tasks. Unlike external cron daemons, an in-process scheduler shares the application’s context: it can access database connections, configuration, and graceful shutdown signals without shell scripts or environment wrangling.

Building a robust scheduling system requires more than parsing cron expressions. You need to handle overlapping executions, distributed locking (when multiple instances run the same schedule), timezone correctness, and observability into what ran, when, and whether it succeeded.

Core Concepts

Key Concept

In-process cron schedulers run inside your application, sharing its lifecycle. When the application shuts down, the scheduler stops gracefully. This is fundamentally different from system cron, which runs independently and has no knowledge of your application’s state.

Key Concept

Extended cron expressions with a seconds field (*/30 * * * * * = every 30 seconds) enable sub-minute scheduling that Unix cron cannot provide. This is essential for health checks, metrics collection, and near-real-time sync jobs.

How It Works

Expression Parsing

A cron expression is parsed into a schedule object that can answer: “given time T, what is the next time this schedule fires?” The parser handles wildcards (*), ranges (1-5), step values (*/15), and lists (MON,WED,FRI). Libraries like robfig/cron compile expressions into bit sets for each field, making next-fire computation a series of fast bitwise operations.

Key Concept

The scheduler does not busy-wait. After dispatching due jobs, it computes the earliest next-fire time across all registered jobs, sets a timer for that instant, and sleeps. This makes idle schedulers essentially free in terms of CPU.

Job Dispatch

When the timer fires, the scheduler iterates over all registered jobs, checks if each job’s next-fire time has passed, and spawns a goroutine (or enqueues to a worker pool) for each due job. The job’s next-fire time is then advanced to the subsequent scheduled instant.

Overlap Prevention

If a job takes longer than its schedule interval, the next invocation may fire while the previous one is still running. Common strategies: (1) skip the overlapping invocation, (2) queue it, or (3) run concurrently. The robfig/cron library provides a DelayIfStillRunning wrapper that serializes overlapping runs and a SkipIfStillRunning wrapper that drops them.

Distributed Scheduling

In a horizontally scaled deployment, every instance runs the same scheduler. Without coordination, a job scheduled at 0 0 * * * would run N times (once per instance). Solutions include: leader election (only the leader runs the scheduler), per-job distributed locks (any instance can win, but only one does), or a dedicated scheduler service.

Timezone Handling

Cron expressions are interpreted relative to a timezone. A job scheduled at 0 0 2 * * * (2:00 AM) in America/New_York must handle DST transitions: the 2:00 AM hour is skipped in spring and repeated in fall. Robust schedulers use timezone-aware time libraries and document their behavior during ambiguous or nonexistent times.

Practical Applications

Implementation Patterns

Code Examples

Go: Basic Cron Scheduler with robfig/cron

package main

import (
    "context"
    "log"
    "os"
    "os/signal"
    "syscall"

    "github.com/robfig/cron/v3"
)

func main() {
    c := cron.New(cron.WithSeconds())

    // Health check every 30 seconds
    c.AddFunc("*/30 * * * * *", func() {
        if err := checkHealth(); err != nil {
            log.Printf("health check failed: %v", err)
        }
    })

    // Data aggregation every hour
    c.AddFunc("0 0 * * * *", func() {
        log.Println("running hourly aggregation")
        if err := aggregateMetrics(context.Background()); err != nil {
            log.Printf("aggregation failed: %v", err)
        }
    })

    // Cleanup at 3 AM daily
    c.AddFunc("0 0 3 * * *", func() {
        log.Println("running daily cleanup")
        cleanupExpiredSessions(context.Background())
    })

    c.Start()
    log.Println("scheduler started")

    // Wait for shutdown signal
    sig := make(chan os.Signal, 1)
    signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
    <-sig

    ctx := c.Stop() // returns context that's done when running jobs finish
    <-ctx.Done()
    log.Println("scheduler stopped gracefully")
}

Go: Distributed Locking Middleware

// DistributedLock wraps a job function so only one instance executes it
func DistributedLock(db *sql.DB, jobName string, fn func()) func() {
    return func() {
        // Try to acquire advisory lock (non-blocking)
        var acquired bool
        err := db.QueryRow(
            "SELECT pg_try_advisory_lock(hashtext($1))", jobName,
        ).Scan(&acquired)
        if err != nil || !acquired {
            return // another instance holds the lock
        }
        defer db.Exec("SELECT pg_advisory_unlock(hashtext($1))", jobName)

        fn()
    }
}

// Usage
c.AddFunc("0 0 * * * *", DistributedLock(db, "hourly-aggregation", func() {
    aggregateMetrics(context.Background())
}))

Go: Job Execution History

func WithHistory(db *sql.DB, jobName string, fn func() error) func() {
    return func() {
        start := time.Now()
        err := fn()
        duration := time.Since(start)

        status := "success"
        var errMsg *string
        if err != nil {
            status = "failed"
            s := err.Error()
            errMsg = &s
        }

        db.Exec(`
            INSERT INTO job_history (job_name, started_at, duration_ms, status, error)
            VALUES ($1, $2, $3, $4, $5)
        `, jobName, start, duration.Milliseconds(), status, errMsg)
    }
}

Common Pitfalls

Key Concepts Recap

Further Reading