Google Kubernetes Engine (GKE) Autopilot is a fully managed Kubernetes mode where Google controls the node lifecycle, scaling, and infrastructure configuration. Unlike Standard GKE where you provision and manage node pools, Autopilot abstracts all node-level concerns—you declare workloads and the control plane provisions appropriately-sized nodes on demand. This dramatically reduces operational overhead while retaining Kubernetes’ full API surface. For teams building SaaS products on GCP, Autopilot is often the right default: it eliminates cluster sprawl, enforces security baselines automatically, and scales to zero during idle periods.
Despite its “fully managed” framing, Autopilot clusters still require careful configuration at provisioning time. The network topology—VPC, subnet, secondary IP ranges for Pods and Services—must be planned before cluster creation because these ranges cannot be changed without recreation. Private cluster mode (nodes without external IPs), Workload Identity for pod-level GCP authentication, managed Prometheus for observability, and DNS-based endpoints for SOC2-compliant control plane access all require explicit configuration. Getting these right at day zero prevents expensive infrastructure migrations later.
In a production cloud-native system, the infrastructure repository manages all GKE cluster provisioning through Pulumi programs written in Go. Each environment (dev, staging, prod) gets its own Autopilot cluster with identical configuration, differing only in subnet CIDR ranges and node resource limits. A SOC2 compliance readiness report documents the specific cluster settings that satisfy control requirements around network isolation, access control, and audit logging. Understanding this provisioning pattern is essential for extending or operating the infrastructure.
GKE requires three distinct CIDR ranges: one for nodes (from the subnet’s primary range), one for Pods (secondary range aliased to --cluster-secondary-range-name), and one for Services (secondary range aliased to --services-secondary-range-name). Using a custom VPC with pre-planned secondary ranges—rather than GKE’s auto-assigned ranges—gives infrastructure teams control over IP space, enables VPC peering without overlap conflicts, and satisfies network segmentation requirements for SOC2. Typical sizing: /14 for Pods (65,536 addresses), /20 for Services (4,096 addresses).
A private GKE cluster configures nodes with only internal IP addresses (no public IPs), and optionally makes the Kubernetes API server endpoint private-only. Private nodes prevent direct inbound access from the internet and satisfy network isolation controls. When enablePrivateEndpoint is true, the API server is only reachable from within the VPC or connected networks—a SOC2 requirement that limits the attack surface on the control plane. Access from developer workstations is provided via Cloud IAP tunneling or a bastion host.
GKE Autopilot supports Spot VMs as a provisioning class for workloads that tolerate interruption. By annotating pods with cloud.google.com/gke-spot: "true", Autopilot provisions spot-priced nodes for those workloads, typically 60–90% cheaper than on-demand. In a typical cloud-native architecture, batch indexing jobs, background workers, and non-critical evaluation pipelines run on Spot nodes while latency-sensitive API pods run on standard nodes. Autopilot handles the node pool separation transparently—the cluster operator only needs to set the provisioning class annotation.
GKE Autopilot can enable Google’s managed Prometheus (GMP) at cluster creation time via managedPrometheus.enabled = true in the monitoring config. GMP deploys a collector DaemonSet that scrapes workloads and forwards metrics to Google Cloud Monitoring, eliminating the need to run and scale a self-managed Prometheus instance. Metrics are stored in Cloud Monitoring with full PromQL query support via the Monarch backend. This satisfies SOC2 monitoring controls out of the box and reduces operational burden for teams without dedicated SRE capacity.
GKE Autopilot supports DNS-based endpoints for the control plane (dnsEndpoint.enabled = true), which replaces the static IP-based endpoint with a stable DNS name. DNS endpoints enable the cluster to be accessed consistently across VPC recreations and simplify firewall rule management since the endpoint resolves within the VPC rather than requiring specific IP allowlists. For SOC2 compliance, DNS endpoints work naturally with Cloud Audit Logs to record every API server access by resolved identity, and they integrate cleanly with Private Service Connect for true VPC-internal routing.
Provisioning a production-grade GKE Autopilot cluster involves coordinating four GCP resource layers: the VPC network with its subnets and secondary ranges, the GKE cluster itself, the Workload Identity binding between Kubernetes service accounts and GCP service accounts, and the monitoring configuration. Pulumi manages all four as a single stack with explicit dependency tracking.
The cluster subnet must be created with secondary IP ranges before the cluster. GKE Autopilot clusters cannot be created in subnets that lack the required secondary ranges, and these ranges cannot be added after cluster creation without destroying the cluster. The infrastructure project pre-plans CIDR allocations across all environments to avoid overlap, using a hierarchical addressing scheme: 10.0.0.0/8 is subdivided by environment and then by purpose (nodes, pods, services).
package infra
import (
"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/compute"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
type NetworkConfig struct {
Region string
NodeCIDR string // primary range for nodes
PodCIDR string // secondary range for pods
ServiceCIDR string // secondary range for services
}
func NewClusterNetwork(ctx *pulumi.Context, env string, cfg NetworkConfig) (*compute.Network, *compute.Subnetwork, error) {
vpc, err := compute.NewNetwork(ctx, env+"-vpc", &compute.NetworkArgs{
Name: pulumi.String(env + "-vpc"),
AutoCreateSubnetworks: pulumi.Bool(false),
RoutingMode: pulumi.String("REGIONAL"),
})
if err != nil {
return nil, nil, err
}
subnet, err := compute.NewSubnetwork(ctx, env+"-subnet", &compute.SubnetworkArgs{
Name: pulumi.String(env + "-gke-subnet"),
Network: vpc.ID(),
Region: pulumi.String(cfg.Region),
IpCidrRange: pulumi.String(cfg.NodeCIDR),
SecondaryIpRanges: compute.SubnetworkSecondaryIpRangeArray{
&compute.SubnetworkSecondaryIpRangeArgs{
RangeName: pulumi.String("pods"),
IpCidrRange: pulumi.String(cfg.PodCIDR),
},
&compute.SubnetworkSecondaryIpRangeArgs{
RangeName: pulumi.String("services"),
IpCidrRange: pulumi.String(cfg.ServiceCIDR),
},
},
PrivateIpGoogleAccess: pulumi.Bool(true),
}, pulumi.DependsOn([]pulumi.Resource{vpc}))
if err != nil {
return nil, nil, err
}
return vpc, subnet, nil
}The container.Cluster resource with enableAutopilot: true is the central declaration. Key fields include the network references, private cluster config, DNS endpoint config, and monitoring config. Autopilot clusters do not accept node pool configurations—those fields are rejected by the API. The cluster creation takes 5–10 minutes and is tracked by Pulumi as a single long-running operation.
package infra
import (
"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/container"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
type ClusterConfig struct {
Project string
Region string
NetworkSelfLink pulumi.StringOutput
SubnetSelfLink pulumi.StringOutput
MasterCIDR string // e.g. "172.16.0.0/28" - must not overlap VPC
}
func NewAutopilotCluster(ctx *pulumi.Context, env string, cfg ClusterConfig) (*container.Cluster, error) {
cluster, err := container.NewCluster(ctx, env+"-cluster", &container.ClusterArgs{
Name: pulumi.String(env + "-cluster"),
Location: pulumi.String(cfg.Region),
Project: pulumi.String(cfg.Project),
EnableAutopilot: pulumi.Bool(true),
Network: cfg.NetworkSelfLink,
Subnetwork: cfg.SubnetSelfLink,
// IP allocation for pods and services from subnet secondary ranges
IpAllocationPolicy: &container.ClusterIpAllocationPolicyArgs{
ClusterSecondaryRangeName: pulumi.String("pods"),
ServicesSecondaryRangeName: pulumi.String("services"),
},
// Private nodes + private control plane endpoint
PrivateClusterConfig: &container.ClusterPrivateClusterConfigArgs{
EnablePrivateNodes: pulumi.Bool(true),
EnablePrivateEndpoint: pulumi.Bool(true),
MasterIpv4CidrBlock: pulumi.String(cfg.MasterCIDR),
},
// DNS-based control plane endpoint (SOC2 compliant)
ControlPlaneEndpointsConfig: &container.ClusterControlPlaneEndpointsConfigArgs{
DnsEndpointConfig: &container.ClusterControlPlaneEndpointsConfigDnsEndpointConfigArgs{
Enabled: pulumi.Bool(true),
},
},
// Managed Prometheus for cluster-wide metrics
MonitoringConfig: &container.ClusterMonitoringConfigArgs{
EnableComponents: pulumi.StringArray{
pulumi.String("SYSTEM_COMPONENTS"),
pulumi.String("WORKLOADS"),
},
ManagedPrometheus: &container.ClusterMonitoringConfigManagedPrometheusArgs{
Enabled: pulumi.Bool(true),
},
},
// Workload Identity for pod-level GCP auth
WorkloadIdentityConfig: &container.ClusterWorkloadIdentityConfigArgs{
WorkloadPool: pulumi.String(cfg.Project + ".svc.id.goog"),
},
// Logging to Cloud Logging
LoggingConfig: &container.ClusterLoggingConfigArgs{
EnableComponents: pulumi.StringArray{
pulumi.String("SYSTEM_COMPONENTS"),
pulumi.String("WORKLOADS"),
},
},
// Deletion protection for production clusters
DeletionProtection: pulumi.Bool(true),
})
if err != nil {
return nil, err
}
return cluster, nil
}In Autopilot, spot provisioning is requested per-workload via a node selector or pod annotation. The infrastructure project includes helper functions that inject the correct annotations into Kubernetes manifests for batch workloads, keeping the cost optimization logic centralized rather than scattered across deployment files.
package k8s
import (
corev1 "k8s.io/api/core/v1"
)
// SpotTolerations returns the tolerations and node selector required
// for GKE Autopilot to provision spot VMs for a workload.
func SpotTolerations() ([]corev1.Toleration, map[string]string) {
tolerations := []corev1.Toleration{
{
Key: "cloud.google.com/gke-spot",
Operator: corev1.TolerationOpEqual,
Value: "true",
Effect: corev1.TaintEffectNoSchedule,
},
}
nodeSelector := map[string]string{
"cloud.google.com/gke-provisioning": "spot",
}
return tolerations, nodeSelector
}
// ApplySpotConfig mutates a PodSpec to request spot provisioning.
func ApplySpotConfig(spec *corev1.PodSpec) {
tolerations, selector := SpotTolerations()
spec.Tolerations = append(spec.Tolerations, tolerations...)
if spec.NodeSelector == nil {
spec.NodeSelector = make(map[string]string)
}
for k, v := range selector {
spec.NodeSelector[k] = v
}
}Even with enablePrivateEndpoint: true, Pulumi and CI/CD pipelines need network-level access to the API server. The infrastructure project configures Master Authorized Networks to allow the VPN gateway CIDR and the Cloud Build service IP ranges. This is set as a separate container.Cluster field rather than post-hoc to avoid a Pulumi diff on every plan.
// Within ClusterArgs, add master authorized networks:
MasterAuthorizedNetworksConfig: &container.ClusterMasterAuthorizedNetworksConfigArgs{
CidrBlocks: container.ClusterMasterAuthorizedNetworksConfigCidrBlockArray{
&container.ClusterMasterAuthorizedNetworksConfigCidrBlockArgs{
CidrBlock: pulumi.String("10.200.0.0/16"), // VPN gateway subnet
DisplayName: pulumi.String("vpn-gateway"),
},
&container.ClusterMasterAuthorizedNetworksConfigCidrBlockArgs{
CidrBlock: pulumi.String("35.199.192.0/19"), // Cloud Build
DisplayName: pulumi.String("cloud-build"),
},
},
},The infrastructure project uses a single Pulumi program with per-stack configuration to provision identical cluster topologies across dev, staging, and production. A ClusterSpec struct is populated from pulumi.Config and drives all resource parameters, ensuring environments only differ where intentional (e.g., CIDR ranges, deletion protection).
package main
import (
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
"platform-infra/infra"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
cfg := config.New(ctx, "")
env := cfg.Require("env") // "dev" | "staging" | "prod"
project := cfg.Require("project")
region := cfg.Require("region")
netCfg := infra.NetworkConfig{
Region: region,
NodeCIDR: cfg.Require("nodeCIDR"),
PodCIDR: cfg.Require("podCIDR"),
ServiceCIDR: cfg.Require("serviceCIDR"),
}
vpc, subnet, err := infra.NewClusterNetwork(ctx, env, netCfg)
if err != nil {
return err
}
clusterCfg := infra.ClusterConfig{
Project: project,
Region: region,
NetworkSelfLink: vpc.SelfLink,
SubnetSelfLink: subnet.SelfLink,
MasterCIDR: cfg.Require("masterCIDR"),
}
cluster, err := infra.NewAutopilotCluster(ctx, env, clusterCfg)
if err != nil {
return err
}
ctx.Export("clusterName", cluster.Name)
ctx.Export("clusterEndpoint", cluster.Endpoint)
return nil
})
}After cluster creation, CI/CD pipelines need a kubeconfig to deploy workloads. The infrastructure project exports cluster details as Pulumi stack outputs and uses a helper that constructs a kubeconfig from the GKE cluster resource, using Application Default Credentials for authentication rather than embedding static tokens.
package infra
import (
"encoding/base64"
"fmt"
"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/container"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
// KubeconfigFromCluster generates a kubeconfig YAML for the cluster.
// Uses gcloud-based exec credential plugin to avoid static tokens.
func KubeconfigFromCluster(cluster *container.Cluster) pulumi.StringOutput {
return pulumi.All(
cluster.Name,
cluster.Endpoint,
cluster.MasterAuth.ClusterCaCertificate(),
).ApplyT(func(args []interface{}) (string, error) {
name := args[0].(string)
endpoint := args[1].(string)
caCert := args[2].(*string)
if caCert == nil {
return "", fmt.Errorf("cluster CA cert not available")
}
_, err := base64.StdEncoding.DecodeString(*caCert)
if err != nil {
return "", fmt.Errorf("invalid CA cert encoding: %w", err)
}
return fmt.Sprintf(`apiVersion: v1
clusters:
- cluster:
certificate-authority-data: %s
server: https://%s
name: %s
contexts:
- context:
cluster: %s
user: %s
name: %s
current-context: %s
kind: Config
users:
- name: %s
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
command: gke-gcloud-auth-plugin
installHint: Install gke-gcloud-auth-plugin for kubectl
provideClusterInfo: true
`, *caCert, endpoint, name, name, name, name, name, name), nil
}).(pulumi.StringOutput)
}Each application service account in Kubernetes needs a corresponding GCP service account binding to access GCS, Pub/Sub, or other GCP APIs without node-level credential sharing. The infrastructure project encapsulates this as a reusable function that creates both the GCP service account and the Kubernetes service account IAM binding.
package infra
import (
"fmt"
"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/iam"
"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/serviceaccount"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
type WorkloadIdentityArgs struct {
Project string
K8sNamespace string
K8sSAName string
GCPRoles []string // e.g. ["roles/storage.objectViewer"]
}
func NewWorkloadIdentity(ctx *pulumi.Context, name string, args WorkloadIdentityArgs) (*serviceaccount.Account, error) {
gcpSA, err := serviceaccount.NewAccount(ctx, name+"-sa", &serviceaccount.AccountArgs{
AccountId: pulumi.String(name + "-sa"),
Project: pulumi.String(args.Project),
DisplayName: pulumi.String(name + " workload identity SA"),
})
if err != nil {
return nil, err
}
// Bind GCP roles to the service account
for i, role := range args.GCPRoles {
_, err = serviceaccount.NewIAMBinding(ctx, fmt.Sprintf("%s-role-%d", name, i), &serviceaccount.IAMBindingArgs{
ServiceAccountId: gcpSA.Name,
Role: pulumi.String(role),
Members: pulumi.StringArray{pulumi.String("serviceAccount:" + args.Project + ".svc.id.goog[" + args.K8sNamespace + "/" + args.K8sSAName + "]")},
})
if err != nil {
return nil, err
}
}
// Allow Kubernetes SA to impersonate GCP SA
_, err = serviceaccount.NewIAMMember(ctx, name+"-wi-binding", &serviceaccount.IAMMemberArgs{
ServiceAccountId: gcpSA.Name,
Role: pulumi.String("roles/iam.workloadIdentityUser"),
Member: pulumi.Sprintf("serviceAccount:%s.svc.id.goog[%s/%s]",
args.Project, args.K8sNamespace, args.K8sSAName),
})
if err != nil {
return nil, err
}
return gcpSA, nil
}In a production infrastructure repository, GKE Autopilot clusters serve as the compute substrate for all platform services. For example, a search API service, a document-indexer (document ingestion), and an image-processing worker all run as Deployments on a shared Autopilot cluster. The cluster’s Autopilot mode means infrastructure engineers never size node pools—workloads declare their CPU and memory requests, and GKE provisions the right nodes automatically. This is particularly valuable for a document-indexer, whose resource usage spikes 10x during large document ingestion jobs and then drops back to near-zero.
A SOC2 readiness report documents how each cluster configuration setting maps to a specific control. Private nodes with private endpoint satisfy CC6.6 (logical access to network boundaries). Workload Identity satisfies CC6.1 (access control for application credentials) by eliminating long-lived service account keys from the cluster. Managed Prometheus and Cloud Logging satisfy CC7.2 (monitoring for security events) by providing immutable audit trails for all control plane and workload activity. The readiness report references specific Pulumi resource field names alongside the control IDs, making it straightforward to verify the infrastructure matches the compliance claims.
For local development, the dev stack uses a smaller master CIDR range and disables deletion protection, but is otherwise identical in configuration to production. This parity is enforced by the parameterized Pulumi program—if a new security feature is added to the production cluster config, it automatically propagates to dev and staging on the next stack update. The infrastructure project’s CI pipeline runs pulumi preview on every PR to catch configuration drift before it reaches production, with the diff posted as a PR comment for review.
masterIpv4CidrBlock must be a /28 that does not overlap with any existing subnet in the VPC or any peered VPC. Autopilot clusters use this range for the managed control plane nodes, and overlap causes cluster creation to fail with an opaque networking error. Reserve /28 blocks in the IP plan before provisioning.privileged: true, hostPath volumes, or hostNetwork will be rejected by the admission webhook. Audit workloads for Autopilot compatibility before migrating from Standard GKE; the most common violations are monitoring agents and log collectors that traditionally run as DaemonSets with host access.gcloud CLI creates state drift that causes Pulumi to generate spurious diffs or, worse, revert intended changes on the next pulumi up. All cluster modifications must go through Pulumi. In a production IaC setup, GCP project IAM permissions for the operations team are set to prevent direct cluster modification, enforcing the IaC contract.