External Secrets Operator with Scoped Workload Identity

Overview

Kubernetes workloads need secrets — database credentials, API keys, TLS certificates — but storing them as native Kubernetes Secrets creates operational problems. Secrets are base64-encoded (not encrypted), scattered across namespaces, and difficult to rotate without redeploying workloads. The External Secrets Operator (ESO) solves this by synchronizing secrets from external providers (GCP Secret Manager, AWS Secrets Manager, HashiCorp Vault) into Kubernetes Secrets, keeping the source of truth in a managed, auditable secret store while making them available to workloads through the standard Kubernetes API.

In the deploy platform, ESO is deployed with a tightly scoped security model. Rather than granting the operator blanket access to all secrets in GCP Secret Manager, the deploy platform uses IAM conditions to restrict ESO to secrets whose names begin with an environment prefix (e.g., prod/ or staging/). The operator’s Kubernetes service account (external-secrets-sa) is bound to a dedicated GCP service account via Workload Identity, ensuring that only pods running as that specific service account can access the scoped secrets. This layered approach — IAM conditions on the GCP side, Workload Identity binding on the Kubernetes side — enforces least-privilege access at every level.

This topic covers the architecture of ESO with scoped Workload Identity, the Helm deployment (including webhook-less mode for development clusters), and the ClusterSecretStore configuration that ties the operator to GCP Secret Manager.

Core Concepts

How It Works

The full ESO deployment in the deploy platform involves four stages: creating the GCP service account with scoped IAM bindings, configuring Workload Identity to link the Kubernetes service account, deploying the ESO Helm chart, and creating the ClusterSecretStore that points ESO at GCP Secret Manager.

Creating the Scoped GCP Service Account

The GCP service account is created with a conditional IAM binding that restricts Secret Manager access to environment-prefixed secrets. The condition uses CEL to match the secret’s resource name against the expected prefix.

// infra/external-secrets.ts
import * as gcp from "@pulumi/gcp";
import * as pulumi from "@pulumi/pulumi";

const env = pulumi.getStack(); // "prod", "staging", etc.

const esoServiceAccount = new gcp.serviceaccount.Account("eso-sa", {
    project: projectId,
    accountId: `eso-${env}`,
    displayName: `External Secrets Operator (${env})`,
});

// Grant Secret Manager access scoped to environment-prefixed secrets.
const secretAccessor = new gcp.projects.IAMMember("eso-secret-accessor", {
    project: projectId,
    role: "roles/secretmanager.secretAccessor",
    member: pulumi.interpolate`serviceAccount:${esoServiceAccount.email}`,
    condition: {
        title: `eso-${env}-scope`,
        description: `Restrict ESO to ${env}-prefixed secrets`,
        expression: `resource.name.startsWith("projects/${projectId}/secrets/${env}-")`,
    },
});

Configuring Workload Identity Binding

The Workload Identity binding connects the Kubernetes service account (external-secrets-sa in the external-secrets namespace) to the GCP service account. This is a two-part configuration: an IAM binding on the GCP side and an annotation on the Kubernetes service account.

// infra/external-secrets.ts (continued)
const workloadIdentityBinding = new gcp.serviceaccount.IAMMember("eso-wi-binding", {
    serviceAccountId: esoServiceAccount.name,
    role: "roles/iam.workloadIdentityUser",
    member: pulumi.interpolate`serviceAccount:${projectId}.svc.id.goog[external-secrets/external-secrets-sa]`,
});

// The Kubernetes service account annotation is set via the Helm chart values:
// serviceAccount:
//   annotations:
//     iam.gke.io/gcp-service-account: eso-prod@my-project.iam.gserviceaccount.com

Deploying the Helm Chart

ESO is deployed via its official Helm chart. The deploy platform passes environment-specific values to configure the service account annotation, webhook behavior, and resource limits.

// infra/external-secrets.ts (continued)
import * as k8s from "@pulumi/kubernetes";

const esoRelease = new k8s.helm.v3.Release("external-secrets", {
    chart: "external-secrets",
    repositoryOpts: {
        repo: "https://charts.external-secrets.io",
    },
    version: "0.9.11",
    namespace: "external-secrets",
    createNamespace: true,
    values: {
        serviceAccount: {
            name: "external-secrets-sa",
            annotations: {
                "iam.gke.io/gcp-service-account":
                    pulumi.interpolate`${esoServiceAccount.email}`,
            },
        },
        webhook: {
            create: env !== "dev", // Disable webhook in dev
        },
        resources: {
            requests: { cpu: "50m", memory: "128Mi" },
            limits:   { cpu: "200m", memory: "256Mi" },
        },
    },
}, { dependsOn: [workloadIdentityBinding] });

Creating the ClusterSecretStore

The ClusterSecretStore tells ESO how to connect to GCP Secret Manager. It references the Kubernetes service account (which, via Workload Identity, authenticates as the GCP service account) and specifies the GCP project where secrets are stored.

// infra/external-secrets.ts (continued)
const clusterSecretStore = new k8s.apiextensions.CustomResource("gcp-secret-store", {
    apiVersion: "external-secrets.io/v1beta1",
    kind: "ClusterSecretStore",
    metadata: {
        name: `gcp-${env}`,
    },
    spec: {
        provider: {
            gcpsm: {
                projectID: projectId,
                auth: {
                    workloadIdentity: {
                        clusterLocation: region,
                        clusterName: clusterName,
                        serviceAccountRef: {
                            name: "external-secrets-sa",
                            namespace: "external-secrets",
                        },
                    },
                },
            },
        },
    },
}, { dependsOn: [esoRelease] });

Using an ExternalSecret

With the ClusterSecretStore in place, workloads reference external secrets via ExternalSecret resources. ESO reconciles each into a native Kubernetes Secret that pods can mount as volumes or inject as environment variables.

# k8s/database-credentials.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-prod
    kind: ClusterSecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: DB_HOST
      remoteRef:
        key: prod-database-host
    - secretKey: DB_PASSWORD
      remoteRef:
        key: prod-database-password
    - secretKey: DB_USERNAME
      remoteRef:
        key: prod-database-username

Practical Applications

The deploy platform uses ESO as the sole mechanism for injecting secrets into Kubernetes workloads. All sensitive configuration — database credentials, API keys, TLS certificates, OAuth client secrets — is stored in GCP Secret Manager and synchronized to Kubernetes via ExternalSecret resources. This eliminates kubectl create secret from deployment pipelines and ensures that secrets are versioned, auditable, and centrally managed.

The environment-scoped IAM conditions provide hard isolation between staging and production secrets. Even if an attacker compromises the staging ESO service account, they cannot read production secrets because the IAM condition is evaluated server-side by GCP. This is a stronger guarantee than namespace-based isolation alone, which relies on Kubernetes RBAC (a single misconfigured ClusterRoleBinding could breach namespace boundaries).

Secret rotation is handled by updating the secret value in GCP Secret Manager. ESO polls the backend at the configured refreshInterval (typically 1 hour for the deploy platform) and updates the Kubernetes Secret when the upstream value changes. Workloads that read secrets from environment variables require a pod restart to pick up changes, but workloads that mount secrets as volumes see updates automatically (Kubernetes propagates volume updates within the kubelet sync period, typically 1 minute).

Typical Integration Points

• Database credentials — PostgreSQL, Redis, and MinIO connection strings synchronized from Secret Manager to application namespaces.
• TLS certificates — Wildcard certificates stored in Secret Manager and synced to Ingress namespaces for TLS termination.
• OAuth client secrets — Identity provider credentials for authentication middleware, rotated in Secret Manager without redeployment.
• Monitoring tokens — Datadog API keys and Grafana Cloud tokens synced to the monitoring namespace.
• Backup encryption keys — AES keys for backup encryption, stored in Secret Manager and synced to the backup CronJob namespace.

Implementation Patterns

Environment Prefix Convention

The deploy platform enforces a naming convention where all secrets in GCP Secret Manager are prefixed with the environment name: prod-database-host, staging-api-key, dev-redis-password. The IAM condition on the ESO service account uses this prefix to scope access. This convention is enforced at the infrastructure layer (Pulumi validates secret names during creation) and at the IAM layer (the condition blocks access to non-matching secrets). The double enforcement prevents both accidental and intentional cross-environment access.

Template-Based Secret Composition

ESO supports templating in the target section, allowing multiple remote secrets to be composed into a single Kubernetes Secret with a custom format. The deploy platform uses this for database connection strings that combine host, port, username, and password into a DSN.

# k8s/database-dsn.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-dsn
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-prod
    kind: ClusterSecretStore
  target:
    name: database-dsn
    template:
      engineVersion: v2
      data:
        DSN: "postgresql://{{ .username }}:{{ .password }}@{{ .host }}:5432/mydb?sslmode=require"
  data:
    - secretKey: username
      remoteRef:
        key: prod-database-username
    - secretKey: password
      remoteRef:
        key: prod-database-password
    - secretKey: host
      remoteRef:
        key: prod-database-host

Refresh Interval Tuning

The refreshInterval controls how often ESO polls the backend for changes. Shorter intervals mean faster secret rotation propagation but higher API call volume (and cost, for backends with per-request pricing). The deploy platform uses 1 hour for most secrets (credentials that rotate infrequently), 15 minutes for TLS certificates (which may be renewed by cert-manager and pushed to Secret Manager), and 5 minutes for feature flag secrets (which change during deployments). Setting the interval to 0 disables polling entirely, requiring manual triggering via annotation changes on the ExternalSecret.

Common Pitfalls

Workload Identity Annotation Mismatch

The Kubernetes service account must have the exact annotation iam.gke.io/gcp-service-account: {gcp-sa-email} for Workload Identity to function. If the annotation references a different GCP service account or is missing entirely, ESO pods will use the GKE node’s default service account (which typically has no Secret Manager access). The error manifests as PERMISSION_DENIED errors in ESO logs, which can be mistaken for an IAM misconfiguration when the root cause is the missing annotation.

IAM Condition CEL Expression Errors

IAM conditions use CEL (Common Expression Language), and syntax errors in the expression are rejected silently — the binding is created but the condition never matches, effectively denying all access. A common mistake is using the wrong resource name format. The correct format for Secret Manager is projects/{project}/secrets/{name}, not projects/{project}/secrets/{name}/versions/latest. Testing conditions with gcloud iam policies lint-condition before deployment catches these errors early.

Forgotten Namespace in Workload Identity Principal

The Workload Identity IAM member format is serviceAccount:{project}.svc.id.goog[{namespace}/{k8s-sa}]. Using the wrong namespace (e.g., default instead of external-secrets) creates a valid binding that never matches the actual ESO pods. This is especially common when the ESO Helm chart creates the namespace automatically and the IAM binding is provisioned in a separate Pulumi stack that hardcodes the namespace name.

Webhook Certificate Issues in Dev Clusters

If the webhook is enabled but cert-manager is not installed, the ESO admission webhook fails to start, and all ExternalSecret create/update operations are rejected with TLS handshake errors. In dev clusters, always set webhook.create: false. In production, ensure cert-manager is deployed and the ESO Helm chart’s webhook.certManager.enabled value is set to true.

Stale Kubernetes Secrets After External Deletion

If a secret is deleted from GCP Secret Manager while an ExternalSecret still references it, ESO logs an error but does not delete the Kubernetes Secret. The stale secret persists with its last-known value until the ExternalSecret is deleted or the secret is recreated in Secret Manager. This can mask secret rotation failures — workloads continue to function with the old value indefinitely. Monitor ESO’s externalsecret_status_condition metric for SecretSyncedError conditions to detect this state.

Further Reading