ConfigMap and Secret Reload Controllers

Cloud & Infrastructure intermediate Building with AI ~30 min read

Overview

Kubernetes gives applications several ways to consume configuration: environment variables, command-line arguments, projected files from ConfigMaps, projected files from Secrets, and external controllers that synthesize those objects from systems such as certificate managers, sealed-secret workflows, or cloud secret stores. The subtle operational problem is that updating the source object does not always update the running process. Volume-mounted files can eventually change on disk, but the application still has to watch and reload them. Environment variables are fixed when the container process starts, so a ConfigMap or Secret update requires a new Pod before the process can observe the new value.

A reload controller closes that gap by watching ConfigMaps and Secrets, finding the workloads that depend on them, and deliberately mutating the workload’s Pod template so Kubernetes performs a normal rolling update. Stakater Reloader is a representative implementation: it watches ConfigMaps, Secrets, and optionally CSI-mounted secrets, then triggers rollouts for Deployments, StatefulSets, DaemonSets, Argo Rollouts, CronJobs, and related workload types. The technique is transferable even if you never use that project: it is a controller pattern for converting data-plane configuration changes into safe workload lifecycle events.

The important design decision is not simply “restart pods when config changes.” It is how to scope the watch, how to discover the affected workloads, how to prevent unrelated changes from cascading through the cluster, and how to let Kubernetes’ native rollout machinery enforce availability constraints. A good reload controller behaves like a small reconciliation engine: observe configuration objects, resolve dependencies, choose eligible workloads, patch only managed metadata, and let the Deployment or StatefulSet controller replace Pods according to the workload’s existing strategy.

Core Concepts

Configuration Propagation Is Not Process Reloading

Kubernetes can update projected ConfigMap and Secret volumes after the backing object changes, but that only changes files in the container filesystem. A process that read its configuration once at startup will continue using stale values unless it watches the files, receives a signal, or restarts. Environment variables are stricter: because they are inherited by the process at launch, Kubernetes cannot update them in place. Replacing the Pod is the standard way to get a fresh process with fresh environment-derived configuration.

Rollout by Pod Template Mutation

Workload controllers create new ReplicaSets or restart Pods when the Pod template changes. Reload controllers exploit that native trigger by patching a harmless annotation on spec.template.metadata.annotations, often with a timestamp, resource version, or hash of the changed object. This avoids deleting Pods directly. The Deployment, StatefulSet, or DaemonSet controller remains responsible for surge limits, unavailable-pod budgets, ordered replacement, readiness gates, and rollback history.

Dependency Discovery

The controller needs a defensible answer to “which workloads should roll out?” One mode inspects Pod templates for envFrom, env.valueFrom, and volume references that point at the changed ConfigMap or Secret. Another mode uses explicit annotations that name the resources to watch even if they are not visible in the Pod spec. A third mode requires both sides to opt in: the workload enables search mode, and the ConfigMap or Secret declares that it is eligible to trigger reloads.

Key Concept: Reload Intent Is a Policy Boundary

A Secret update may be security-critical, harmless, or experimental. Automatically rolling every consumer can cause needless churn, while never rolling consumers can leave revoked credentials in memory. Annotation-based policy lets platform teams encode intent per workload and per configuration object instead of treating every update as globally equivalent.

How It Works

Watch, Resolve, Patch

The controller subscribes to ConfigMap and Secret update events. For each change, it lists candidate workloads in the namespace, filters them by annotations and references, then patches each selected Pod template. The patch should be idempotent, narrow, and owned by the controller so it does not overwrite application-managed annotations.

// Simplified controller flow. Real controllers also handle namespaces,
// RBAC failures, rate limiting, retries, and workload-specific APIs.
func onConfigChanged(ctx context.Context, changed ObjectRef) error {
    workloads, err := listWorkloads(ctx, changed.Namespace)
    if err != nil {
        return err
    }

    for _, workload := range workloads {
        policy := reloadPolicy(workload)
        if !policy.Enabled {
            continue
        }
        if !matchesChangedObject(workload.PodTemplate, policy, changed) {
            continue
        }

        patch := map[string]string{
            "reloader.example.com/last-reloaded-at": time.Now().UTC().Format(time.RFC3339),
            "reloader.example.com/last-reloaded-from": changed.Kind + "/" + changed.Name,
        }
        if err := patchPodTemplateAnnotations(ctx, workload, patch); err != nil {
            return err
        }
    }
    return nil
}

Annotation Modes

A practical reload controller usually supports multiple selection modes because teams have different safety requirements. Automatic mode reloads when a workload references any changed ConfigMap or Secret. Named-resource mode reloads only when a specific configured name changes. Match-and-search mode requires the workload to search for eligible resources and the resource itself to opt in. Exclusion annotations are also useful for shared ConfigMaps that are mounted by many Pods but should not trigger every consumer.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  annotations:
    reloader.example.com/auto: "true"
spec:
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
      - name: app
        image: example/checkout-api:1.8.4
        envFrom:
        - configMapRef:
            name: checkout-config
        - secretRef:
            name: checkout-secrets

Why Patch the Workload Instead of the Pod?

Deleting or patching individual Pods fights the owning controller. A Deployment will replace deleted Pods, but it loses the semantic link between the config change and the rollout history. Patching the Pod template creates an auditable rollout event, respects maxUnavailable and maxSurge, and gives teams the same observability path they use for image updates. For StatefulSets, this distinction is even more important because ordered replacement and stable identities are part of the workload contract.

Practical Applications

Credential Rotation

Applications often read database passwords, API tokens, signing keys, or certificate paths once at startup. When an external secret manager updates the Kubernetes Secret, the Pod may keep using the old credential until it restarts. A reload controller makes secret rotation operationally reliable by tying Secret updates to controlled workload rollouts. This is especially useful when the secret is generated by another controller, such as an external-secret synchronizer or certificate issuer.

Feature Flags and Runtime Configuration

Some applications can reload feature flags from a file watcher, but many simpler services read configuration at boot. For those services, a ConfigMap update without a Pod replacement creates a split-brain operational state: Kubernetes stores the desired setting, while live processes keep the old setting. Reload controllers remove that ambiguity by making the rollout explicit and visible.

GitOps and CI/CD Pipelines

GitOps pipelines frequently apply ConfigMap and Secret changes independently from image changes. If the workload manifest itself does not change, Kubernetes has no reason to create a new ReplicaSet. A reload controller lets configuration-only commits still produce a rollout without requiring every chart or kustomization to compute checksum annotations manually.

Implementation Patterns

Use Informers and Work Queues

A production controller should not perform heavy work directly inside the watch callback. Watch callbacks enqueue keys, and workers process those keys with retry and backoff. This keeps event handling responsive, deduplicates bursts of rapid updates, and lets the controller recover from transient API failures.

type QueueItem struct {
    Namespace string
    Kind      string
    Name      string
    Version   string
}

func enqueueConfigMapUpdate(oldObj, newObj *corev1.ConfigMap, q workqueue.RateLimitingInterface) {
    if oldObj.ResourceVersion == newObj.ResourceVersion {
        return
    }
    q.Add(QueueItem{
        Namespace: newObj.Namespace,
        Kind:      "ConfigMap",
        Name:      newObj.Name,
        Version:   newObj.ResourceVersion,
    })
}

Patch a Stable Annotation Contract

The annotation value should change whenever a rollout is required, but the key should remain stable. Teams commonly store the last reload timestamp, the triggering resource name, or a digest of the relevant configuration. Digests are useful when you want identical data to be a no-op; timestamps are useful when the source event itself should force a replacement even if the bytes are the same.

Separate Detection from Authorization

Detection answers whether a workload references a resource. Authorization answers whether that reference is allowed to trigger a rollout. Keeping those decisions separate makes the controller easier to audit. For example, a shared trust bundle mounted into hundreds of workloads might be detected everywhere, but only workloads annotated for automatic reload should be patched.

Performance Considerations

Common Pitfalls

Further Reading