Database backups are a non-negotiable requirement for any production system. While manual backups are possible, they introduce human error and gaps in coverage. Kubernetes CronJobs provide a declarative, schedulable mechanism for running backup tasks at fixed intervals, and when combined with a custom reconciler, they become a fully automated backup pipeline that generates jobs, injects credentials, enforces deadlines, and labels everything for traceability.
The DatabaseBackup custom resource in database-operator encapsulates this pattern. Rather than hand-crafting CronJob YAML for each database cluster, operators declare a backup schedule, retention policy, and target storage location in an DatabaseBackup CR. The reconciler then generates the corresponding Kubernetes CronJob that hits the cluster’s public API to trigger a snapshot, mounts credential secrets for S3 access, and enforces time windows so stale backup jobs don’t accumulate.
This topic covers how the DatabaseBackup reconciler works, the Kubernetes CronJob mechanics it relies on, and the patterns for building reliable scheduled backup systems in Kubernetes. Understanding this integration is essential for anyone operating exampledb clusters in production or building similar operator-driven backup automation.
A CronJob is a Kubernetes resource that creates Job objects on a repeating schedule defined by a cron expression. Each Job spawns one or more Pods that run to completion. CronJobs support concurrency policies (Allow, Forbid, Replace), deadline enforcement via startingDeadlineSeconds, and history limits that control how many successful and failed Jobs to retain. The CronJob controller in kube-controller-manager evaluates schedules every 10 seconds and creates Jobs when the next occurrence is due.
A Kubernetes reconciler watches a custom resource (CR) and continuously drives the cluster state toward the desired state declared in the CR’s spec. For DatabaseBackup, the reconciler watches DatabaseBackup CRs and ensures a corresponding CronJob exists with the correct schedule, container image, environment variables, and volume mounts. If the CR is updated (e.g., schedule changed from hourly to every 6 hours), the reconciler updates the CronJob. If the CronJob is deleted externally, the reconciler recreates it on the next reconciliation loop.
Backup jobs need credentials to access both the database (API token or mTLS certificate) and the storage backend (S3 access keys). Rather than embedding credentials in the CronJob spec, the reconciler references Kubernetes Secrets by name. The backup Pod mounts these secrets as environment variables or files, ensuring credentials are never stored in the CronJob definition and can be rotated independently without redeploying the backup configuration.
The startingDeadlineSeconds field prevents the CronJob controller from creating a Job if it missed the scheduled time by more than the configured window (e.g., due to cluster downtime). successfulJobsHistoryLimit and failedJobsHistoryLimit control how many completed Job objects are retained, preventing unbounded growth of Job metadata in etcd. The DatabaseBackup reconciler sets sensible defaults (deadline of 300 seconds, keep 3 successful and 1 failed) but allows overrides in the CR spec.
Every CronJob, Job, and Pod generated by the reconciler carries labels linking it back to the originating DatabaseBackup CR: example.com/backup-name, example.com/cluster-name, and example.com/backup-schedule. These labels enable operators to query all backup-related resources for a specific cluster (kubectl get jobs -l example.com/cluster-name=prod-us-east), integrate with monitoring systems that alert on failed Jobs, and implement garbage collection for orphaned resources.
The DatabaseBackup reconciler follows a standard controller-runtime pattern. When an DatabaseBackup CR is created or updated, the reconciler builds the desired CronJob spec from the CR’s fields, compares it against the existing CronJob (if any), and applies a server-side patch to converge the state. The reconciler also manages the Secret references and validates that the target cluster exists before creating the CronJob.
The CR declares the backup schedule, target cluster, storage configuration, and retention policy. The reconciler interprets these fields and generates the corresponding CronJob.
// api/v1alpha1/databasebackup_types.go
package v1alpha1
import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
// DatabaseBackupSpec defines the desired state of DatabaseBackup.
type DatabaseBackupSpec struct {
// ClusterName is the name of the ExampleDB cluster to back up.
ClusterName string `json:"clusterName"`
// Schedule is a cron expression (e.g., "0 */6 * * *" for every 6 hours).
Schedule string `json:"schedule"`
// Storage defines where backups are stored.
Storage BackupStorageSpec `json:"storage"`
// CredentialSecretRef points to a Secret containing API and S3 credentials.
CredentialSecretRef string `json:"credentialSecretRef"`
// StartingDeadlineSeconds is the optional deadline for starting a backup job.
StartingDeadlineSeconds *int64 `json:"startingDeadlineSeconds,omitempty"`
// RetentionPolicy defines how many backups to keep.
RetentionPolicy RetentionPolicySpec `json:"retentionPolicy,omitempty"`
}
type BackupStorageSpec struct {
S3Endpoint string `json:"s3Endpoint"`
Bucket string `json:"bucket"`
Prefix string `json:"prefix,omitempty"`
}
type RetentionPolicySpec struct {
MaxBackups int `json:"maxBackups,omitempty"`
MaxAgeDays int `json:"maxAgeDays,omitempty"`
}
type DatabaseBackup struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec DatabaseBackupSpec `json:"spec,omitempty"`
Status DatabaseBackupStatus `json:"status,omitempty"`
}
type DatabaseBackupStatus struct {
LastSuccessfulBackup *metav1.Time `json:"lastSuccessfulBackup,omitempty"`
LastScheduleTime *metav1.Time `json:"lastScheduleTime,omitempty"`
Active []string `json:"active,omitempty"`
}The reconciler constructs a CronJob whose Pod template runs a backup container. The container calls the exampledb cluster’s public HTTP API to trigger a snapshot export, then streams the result to S3. All configuration is injected via environment variables sourced from the credential Secret and the CR spec.
// controllers/databasebackup_controller.go
package controllers
import (
"context"
"fmt"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/utils/ptr"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
v1alpha1 "github.com/example/database-operator/api/v1alpha1"
)
func (r *DatabaseBackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
log := ctrl.LoggerFrom(ctx)
var backup v1alpha1.DatabaseBackup
if err := r.Get(ctx, req.NamespacedName, &backup); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
cronJob := r.buildCronJob(&backup)
// Set the DatabaseBackup as the owner so garbage collection works.
if err := ctrl.SetControllerReference(&backup, cronJob, r.Scheme); err != nil {
return ctrl.Result{}, fmt.Errorf("setting controller reference: %w", err)
}
// Server-side apply to converge state.
if err := r.Patch(ctx, cronJob, client.Apply, client.FieldOwner("database-backup-controller")); err != nil {
return ctrl.Result{}, fmt.Errorf("applying CronJob: %w", err)
}
log.Info("reconciled backup CronJob",
"cluster", backup.Spec.ClusterName,
"schedule", backup.Spec.Schedule,
)
return ctrl.Result{}, nil
}
func (r *DatabaseBackupReconciler) buildCronJob(backup *v1alpha1.DatabaseBackup) *batchv1.CronJob {
labels := map[string]string{
"example.com/backup-name": backup.Name,
"example.com/cluster-name": backup.Spec.ClusterName,
"example.com/backup-schedule": backup.Spec.Schedule,
"app.kubernetes.io/part-of": "exampledb",
}
deadline := ptr.To(int64(300))
if backup.Spec.StartingDeadlineSeconds != nil {
deadline = backup.Spec.StartingDeadlineSeconds
}
return &batchv1.CronJob{
TypeMeta: metav1.TypeMeta{APIVersion: "batch/v1", Kind: "CronJob"},
ObjectMeta: metav1.ObjectMeta{
Name: fmt.Sprintf("%s-backup", backup.Name),
Namespace: backup.Namespace,
Labels: labels,
},
Spec: batchv1.CronJobSpec{
Schedule: backup.Spec.Schedule,
ConcurrencyPolicy: batchv1.ForbidConcurrent,
StartingDeadlineSeconds: deadline,
SuccessfulJobsHistoryLimit: ptr.To(int32(3)),
FailedJobsHistoryLimit: ptr.To(int32(1)),
JobTemplate: batchv1.JobTemplateSpec{
ObjectMeta: metav1.ObjectMeta{Labels: labels},
Spec: batchv1.JobSpec{
ActiveDeadlineSeconds: ptr.To(int64(1800)),
BackoffLimit: ptr.To(int32(2)),
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{Labels: labels},
Spec: corev1.PodSpec{
RestartPolicy: corev1.RestartPolicyOnFailure,
Containers: []corev1.Container{{
Name: "backup",
Image: "exampledb/backup-agent:latest",
EnvFrom: []corev1.EnvFromSource{{
SecretRef: &corev1.SecretEnvSource{
LocalObjectReference: corev1.LocalObjectReference{
Name: backup.Spec.CredentialSecretRef,
},
},
}},
Env: []corev1.EnvVar{
{Name: "CLUSTER_NAME", Value: backup.Spec.ClusterName},
{Name: "S3_ENDPOINT", Value: backup.Spec.Storage.S3Endpoint},
{Name: "S3_BUCKET", Value: backup.Spec.Storage.Bucket},
{Name: "S3_PREFIX", Value: backup.Spec.Storage.Prefix},
},
}},
},
},
},
},
},
}
}The backup agent is a small Go binary that runs inside the Job Pod. It calls the exampledb cluster’s admin API to initiate a snapshot, streams the response body directly to S3 using multipart upload, and exits with a non-zero code on failure so Kubernetes can retry according to the Job’s backoff policy.
// cmd/backup-agent/main.go
package main
import (
"context"
"fmt"
"log"
"net/http"
"os"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
clusterName := os.Getenv("CLUSTER_NAME")
apiURL := fmt.Sprintf("http://%s.database-system.svc:8080/admin/snapshot", clusterName)
// Trigger the snapshot via the cluster's admin API.
resp, err := http.Get(apiURL)
if err != nil {
log.Fatalf("triggering snapshot: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
log.Fatalf("snapshot API returned %d", resp.StatusCode)
}
// Upload the snapshot stream directly to S3.
cfg, _ := config.LoadDefaultConfig(ctx,
config.WithRegion(os.Getenv("AWS_REGION")),
config.WithCredentialsProvider(
credentials.NewStaticCredentialsProvider(
os.Getenv("AWS_ACCESS_KEY_ID"),
os.Getenv("AWS_SECRET_ACCESS_KEY"), "",
),
),
)
s3Client := s3.NewFromConfig(cfg, func(o *s3.Options) {
if ep := os.Getenv("S3_ENDPOINT"); ep != "" {
o.BaseEndpoint = aws.String(ep)
o.UsePathStyle = true
}
})
key := fmt.Sprintf("%s/%s/snapshot-%s.bin",
os.Getenv("S3_PREFIX"), clusterName,
time.Now().UTC().Format("20060102-150405"),
)
uploader := manager.NewUploader(s3Client)
_, err = uploader.Upload(ctx, &s3.PutObjectInput{
Bucket: aws.String(os.Getenv("S3_BUCKET")),
Key: aws.String(key),
Body: resp.Body,
})
if err != nil {
log.Fatalf("uploading snapshot to S3: %v", err)
}
log.Printf("backup completed: s3://%s/%s", os.Getenv("S3_BUCKET"), key)
}In the database-operator ecosystem, the DatabaseBackup CR is the primary mechanism for ensuring every production cluster has automated backup coverage. When a new ExampleDB cluster is provisioned, the operator automatically creates a companion DatabaseBackup CR with sensible defaults (every 6 hours, 7-day retention, same S3 bucket as the cluster’s snapshot store). Operators can override these defaults or create additional backup schedules for compliance requirements — for example, a daily backup retained for 30 days and a weekly backup retained for 90 days.
The label-based traceability is particularly valuable for multi-tenant deployments where dozens of clusters are running in the same Kubernetes namespace. Platform teams can query all backup Jobs for a specific cluster, identify failed backups across all clusters, and feed Job status into Prometheus via kube-state-metrics to power alerting rules like “alert if no successful backup for cluster X in 24 hours.”
The CronJob’s concurrency policy of Forbid prevents overlapping backup Jobs, which is critical because exampledb’s snapshot API is a heavyweight operation that temporarily increases memory pressure on the leader node. If a backup takes longer than the schedule interval, the CronJob controller simply skips the next occurrence rather than spawning a concurrent backup that could destabilize the cluster.
lastSuccessfulBackup, active) are exposed as Prometheus metrics for audit dashboards.The reconciler sets the DatabaseBackup CR as the owner of the generated CronJob using ctrl.SetControllerReference. This means deleting the DatabaseBackup CR automatically cascades to delete the CronJob, preventing orphaned resources. The owner reference also enables the controller-runtime framework to trigger reconciliation when the owned CronJob changes (e.g., if someone manually edits or deletes it).
Instead of using Create/Update with optimistic locking, the reconciler uses server-side apply (client.Apply with a field owner). This approach is idempotent — applying the same desired state repeatedly is a no-op — and handles field conflicts gracefully. If another controller modifies the CronJob, the next reconciliation restores the fields owned by the backup controller without stomping on fields managed by other controllers.
For teams that prefer plain YAML over the operator, the same pattern can be expressed as a raw CronJob with environment variables sourced from Secrets:
apiVersion: batch/v1
kind: CronJob
metadata:
name: exampledb-prod-backup
labels:
example.com/cluster-name: prod-us-east
example.com/backup-schedule: "0 */6 * * *"
spec:
schedule: "0 */6 * * *"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
jobTemplate:
spec:
activeDeadlineSeconds: 1800
backoffLimit: 2
template:
spec:
restartPolicy: OnFailure
containers:
- name: backup
image: exampledb/backup-agent:latest
envFrom:
- secretRef:
name: exampledb-prod-backup-creds
env:
- name: CLUSTER_NAME
value: prod-us-east
- name: S3_BUCKET
value: exampledb-backups-prod
- name: S3_PREFIX
value: backupsWithout startingDeadlineSeconds, the CronJob controller accumulates missed schedules after a cluster outage and fires them all at once when the controller recovers. For a backup that runs every hour, a 12-hour outage could trigger 12 simultaneous backup Jobs, overwhelming both the database and the S3 endpoint. Always set a starting deadline (e.g., 300 seconds) so missed schedules are skipped rather than replayed.
The default successfulJobsHistoryLimit is 3 and failedJobsHistoryLimit is 1, but if not explicitly set, some Kubernetes versions default to keeping all Jobs. In a long-running cluster with hourly backups, this can create thousands of completed Job objects in etcd, increasing API server memory usage and slowing kubectl get jobs queries. Always set explicit history limits.
When S3 credentials are rotated in the referenced Secret, existing in-flight backup Jobs continue using the old credentials (they were injected at Pod creation time). If the old credentials are immediately revoked, the in-flight Job fails. The safer pattern is to keep old credentials valid for a grace period longer than the Job’s activeDeadlineSeconds, or use IAM roles with Workload Identity instead of static credentials.
Automated backups create a false sense of security if the restore path is never tested. A common failure mode is that backups succeed but are silently corrupted (e.g., an incomplete multipart upload that was marked successful due to a client bug). Periodically run restore tests that download the latest backup, load it into a temporary cluster, and run a data integrity check. This can itself be a CronJob on a less frequent schedule.
Backup agent Pods that don’t specify resource requests and limits may be evicted under memory pressure or starve other workloads. For large database snapshots, the backup agent streams data rather than buffering it in memory, but the S3 multipart uploader still allocates per-part buffers. Set memory limits based on partSize * concurrency (e.g., 64MB parts with concurrency 4 = 256MB plus overhead).