External Secrets Operator (ESO)

Overview

The External Secrets Operator (ESO) is a Kubernetes operator that synchronizes secrets from external secret management systems—such as Google Secret Manager, AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault—into native Kubernetes Secrets. Instead of manually creating kubectl create secret entries or embedding secrets in Helm values, teams declare an ExternalSecret custom resource that references a secret in the external store. ESO’s controller watches these resources, fetches the secret values, and creates or updates the corresponding Kubernetes Secret automatically.

This pattern solves a fundamental tension in Kubernetes secret management: applications need secrets as Kubernetes Secret objects (mounted as volumes or environment variables), but storing secret values in Git, Helm charts, or CI/CD pipelines violates security best practices and SOC2 controls. ESO bridges this gap by keeping the secret source of truth in a purpose-built secret manager while delivering secrets to workloads in the format they expect.

In a typical production Kubernetes environment, teams deploy ESO via Helm and configure ClusterSecretStore resources that authenticate to Google Secret Manager using Workload Identity. Each environment (dev, staging, prod) has its own Secret Manager project, and ESO syncs database credentials, API keys, webhook secrets, and TLS certificates into the appropriate namespaces. This pattern helps satisfy SOC2 controls for secret rotation, access control, and audit logging of secret access.

Core Concepts

ClusterSecretStore vs SecretStore

A SecretStore is namespace-scoped—it can only be referenced by ExternalSecret resources in the same namespace. A ClusterSecretStore is cluster-scoped and can be referenced from any namespace. In practice, teams use ClusterSecretStore for the shared Google Secret Manager connection when secrets are consumed across multiple namespaces (search, pipeline, billing, etc.). This reduces configuration duplication while still allowing per-namespace access control through the ExternalSecret resources themselves.

ExternalSecret Resource

An ExternalSecret declares which secrets to fetch, from which store, and how to map them into a Kubernetes Secret. Each ExternalSecret specifies a secretStoreRef (pointing to a ClusterSecretStore or SecretStore), a target (the name of the Kubernetes Secret to create), and a data array mapping remote secret keys to local secret keys. The controller reconciles this resource on a configurable interval (e.g., every 1 hour), automatically picking up rotated values from the external store.

IAM Conditions for Granular Access Control

When using Google Secret Manager with ESO, IAM conditions can restrict which secrets the ESO service account can access. Rather than granting roles/secretmanager.secretAccessor on the entire project, the infrastructure project uses conditional IAM bindings that scope access to secrets matching a resource name pattern (e.g., resource.name.startsWith("projects/PROJECT/secrets/prod-")). This prevents the prod ESO instance from reading dev secrets and vice versa, even though they share the same project.

Refresh Interval and Secret Rotation

ESO’s refreshInterval controls how often the controller re-fetches secrets from the external store. A short interval (e.g., 5 minutes) ensures rotated credentials are picked up quickly but increases API calls. A long interval (e.g., 24 hours) is cheaper but delays rotation propagation. A production setup uses 1-hour intervals for database credentials (which rotate quarterly) and 5-minute intervals for Stripe webhook signing secrets (which may be rotated during incident response).

Helm-Based Deployment

ESO is deployed via its official Helm chart, which installs the CRDs (ExternalSecret, SecretStore, ClusterSecretStore, ClusterExternalSecret), the controller deployment, the webhook for validation, and the cert-controller for webhook TLS. Pulumi’s Helm release resource manages the installation, pinning to a specific chart version and passing values that configure the controller’s service account for Workload Identity authentication.

How It Works

Setting up ESO for Google Secret Manager involves four steps: deploying the operator via Helm, creating a ClusterSecretStore that authenticates to Secret Manager, defining ExternalSecret resources for each secret, and verifying the sync status.

Deploying ESO via Helm with Pulumi

The infrastructure project manages the ESO Helm release as a Pulumi resource, ensuring the operator version is pinned and the service account is configured for Workload Identity. The Helm chart creates the CRDs, controller, webhook, and cert-controller in a dedicated namespace.

package k8s

import (
    helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
    corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
    metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func DeployESO(ctx *pulumi.Context, gsaEmail string) error {
    ns, err := corev1.NewNamespace(ctx, "external-secrets", &corev1.NamespaceArgs{
        Metadata: &metav1.ObjectMetaArgs{
            Name: pulumi.String("external-secrets"),
        },
    })
    if err != nil {
        return err
    }

    _, err = helmv3.NewRelease(ctx, "external-secrets", &helmv3.ReleaseArgs{
        Name:      pulumi.String("external-secrets"),
        Namespace: ns.Metadata.Name(),
        Chart:     pulumi.String("external-secrets"),
        Version:   pulumi.String("0.9.11"),
        RepositoryOpts: &helmv3.RepositoryOptsArgs{
            Repo: pulumi.String("https://charts.external-secrets.io"),
        },
        Values: pulumi.Map{
            "serviceAccount": pulumi.Map{
                "annotations": pulumi.Map{
                    "iam.gke.io/gcp-service-account": pulumi.String(gsaEmail),
                },
            },
            "webhook": pulumi.Map{
                "port": pulumi.Int(9443),
            },
        },
    })
    return err
}

Creating a ClusterSecretStore for Google Secret Manager

The ClusterSecretStore tells ESO how to authenticate to Google Secret Manager. With Workload Identity, no explicit credentials are needed—ESO’s service account is bound to a GSA with Secret Manager access, and the GKE metadata server handles token exchange transparently.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-manager
spec:
  provider:
    gcpsm:
      projectID: "mydb-prod"
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: prod-cluster
          clusterProjectID: "mydb-prod"
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

Defining ExternalSecret Resources

Each secret that workloads need is declared as an ExternalSecret. The resource specifies which ClusterSecretStore to use, the refresh interval, and the mapping from Secret Manager secret names to Kubernetes Secret data keys.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: search
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-manager
    kind: ClusterSecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: DB_HOST
      remoteRef:
        key: prod-db-host
    - secretKey: DB_PASSWORD
      remoteRef:
        key: prod-db-password
    - secretKey: DB_USERNAME
      remoteRef:
        key: prod-db-username

Provisioning ExternalSecrets via Pulumi

The infrastructure project creates ExternalSecret resources programmatically in Pulumi rather than applying raw YAML. This allows secrets to be parameterized by environment and namespace, with the secret key prefix (e.g., prod- vs dev-) derived from the stack configuration.

package k8s

import (
    "fmt"

    apiextv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

type SecretMapping struct {
    LocalKey  string // key in the Kubernetes Secret
    RemoteKey string // key in Google Secret Manager
}

func NewExternalSecret(ctx *pulumi.Context, env, namespace, name string, mappings []SecretMapping) error {
    data := make([]map[string]interface{}, len(mappings))
    for i, m := range mappings {
        data[i] = map[string]interface{}{
            "secretKey": m.LocalKey,
            "remoteRef": map[string]interface{}{
                "key": fmt.Sprintf("%s-%s", env, m.RemoteKey),
            },
        }
    }

    _, err := apiextv1.NewCustomResource(ctx,
        fmt.Sprintf("%s-%s-es", namespace, name),
        &apiextv1.CustomResourceArgs{
            ApiVersion: pulumi.String("external-secrets.io/v1beta1"),
            Kind:       pulumi.String("ExternalSecret"),
            Metadata: pulumi.Map{
                "name":      pulumi.String(name),
                "namespace": pulumi.String(namespace),
            },
            OtherFields: map[string]interface{}{
                "spec": map[string]interface{}{
                    "refreshInterval": "1h",
                    "secretStoreRef": map[string]interface{}{
                        "name": "gcp-secret-manager",
                        "kind": "ClusterSecretStore",
                    },
                    "target": map[string]interface{}{
                        "name":           name,
                        "creationPolicy": "Owner",
                    },
                    "data": data,
                },
            },
        },
    )
    return err
}

Implementation Patterns

Environment-Prefixed Secret Keys

The infrastructure project prefixes all Secret Manager secret names with the environment (e.g., prod-db-password, dev-db-password). The Pulumi stack configuration determines which prefix to use, so the same ExternalSecret template works across environments. This prevents accidental cross-environment secret access and makes IAM conditions straightforward: the prod ESO service account can only access secrets starting with prod-.

Owner Creation Policy for Cleanup

Setting creationPolicy: Owner on the target means the Kubernetes Secret is owned by the ExternalSecret resource. Deleting the ExternalSecret automatically garbage-collects the Kubernetes Secret, preventing orphaned secrets from accumulating. This is critical for namespace teardowns during environment reprovisioning.

Monitoring Sync Status

ESO exposes the sync status on each ExternalSecret’s .status.conditions. A production setup includes a Prometheus alert rule that fires when any ExternalSecret has been in a SecretSyncedError state for more than 15 minutes, catching authentication failures, deleted upstream secrets, or API quota exhaustion before they impact workloads.

groups:
  - name: external-secrets
    rules:
      - alert: ExternalSecretSyncFailed
        expr: |
          externalsecret_status_condition{condition="Ready", status="False"} == 1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "ExternalSecret {{ $labels.name }} in {{ $labels.namespace }} failed to sync"
          description: "The ExternalSecret has been in error state for 15+ minutes. Check ESO controller logs and Secret Manager permissions."

Practical Applications

ESO is well-suited for deployment across a Kubernetes infrastructure for secret management. For example:

Database Credentials: Cloud SQL connection strings, usernames, and passwords are stored in Secret Manager and synced to the search, pipeline, and billing namespaces. When database credentials are rotated, updating the Secret Manager value triggers automatic propagation within the refresh interval.
Stripe API Keys: The billing service consumes Stripe publishable and secret keys, plus webhook signing secrets. These are synced with a 5-minute refresh interval to support emergency rotation during payment processing incidents.
TLS Certificates: Internal service-to-service mTLS certificates stored in Secret Manager are synced as Kubernetes TLS secrets and mounted by Envoy sidecars.
Feature Flags Service Token: The application’s feature flag service API token is managed through ESO rather than hardcoded in deployment manifests, enabling rotation without redeployment.

Common Pitfalls

CRD version mismatch: Upgrading ESO without upgrading CRDs (or vice versa) causes validation webhook failures. Always use the Helm chart’s built-in CRD management rather than applying CRDs separately.
Workload Identity not propagated: If the ESO controller pod starts before the Workload Identity binding is active, it caches the authentication failure. Restart the controller after verifying the WI binding is correct.
Secret Manager API quota: With many ExternalSecret resources and short refresh intervals, ESO can exhaust the Secret Manager API quota (1,800 requests/minute by default). Monitor quota usage and increase the refresh interval for non-critical secrets.
Missing secret versions: If a Secret Manager secret is disabled or destroyed, ESO logs an error but keeps the last-synced Kubernetes Secret intact. This is by design (preventing secret deletion from crashing workloads) but means stale secrets can persist silently.
Namespace-scoped vs cluster-scoped confusion: Using a SecretStore in one namespace and trying to reference it from another namespace fails silently. Default to ClusterSecretStore unless you need namespace-level authentication isolation.