GitHub Actions Runner Controller (ARC) on Kubernetes

Cloud & Infrastructure intermediate Building with AI ~20 min read

Overview

GitHub-hosted runners work well for simple CI pipelines, but production infrastructure often demands more: custom toolchains, GPU access, network-local artifact caches, or the ability to scale from zero during off-hours and burst to dozens of runners during peak deployment windows. GitHub Actions Runner Controller (ARC) bridges this gap by running self-hosted GitHub Actions runners as Kubernetes pods, managed by a custom controller that watches for pending workflow jobs and scales runner replicas accordingly.

In a cloud-native project’s infrastructure repository, ARC is deployed on GKE with multiple runner profiles — standard runners for lint and unit tests, heavy runners with more CPU and memory for integration test suites, and docker-enabled runners that can build container images using Docker-in-Docker (DinD). Each profile maps to a separate RunnerDeployment with its own resource requests, node affinity rules, and scale-to-zero configuration. GitHub App authentication credentials are managed through ExternalSecrets, pulling from GCP Secret Manager so that no long-lived tokens are stored in cluster manifests.

This architecture eliminates the fixed cost of always-on runners while providing consistent, reproducible build environments that match production infrastructure. When no workflows are pending, the cluster scales to zero runner pods; when a developer pushes to a PR, ARC detects the queued job and spins up a runner within seconds.

Core Concepts

Runner Deployments and Scale Sets

ARC introduces the RunnerDeployment custom resource, which defines a template for runner pods — including container image, resource limits, labels, and volume mounts. A HorizontalRunnerAutoscaler (HRA) watches GitHub’s API for pending workflow runs that match the runner’s labels and adjusts the replica count between a configured minimum (typically 0) and maximum. Scale sets group runners by capability so that workflows requesting runs-on: heavy only trigger scaling of heavy-profile runners.

Runner Profiles

Different CI workloads have different resource requirements. A “standard” profile might request 2 CPU cores and 4 GiB RAM for linting and unit tests, while a “heavy” profile requests 8 cores and 16 GiB for compilation-heavy integration tests. A “docker” profile adds a Docker-in-Docker sidecar container for building container images. Each profile is a separate RunnerDeployment with its own autoscaler, node pool affinity, and GitHub runner labels.

GitHub App Authentication

ARC authenticates with GitHub using a GitHub App installation rather than personal access tokens. The App’s private key and installation ID are stored in GCP Secret Manager and synced into Kubernetes secrets via the External Secrets Operator. This avoids embedding long-lived credentials in Helm values or Terraform state, and the GitHub App’s fine-grained permissions can be scoped to only the repositories that need self-hosted runners.

Scale-to-Zero Economics

By setting the minimum replica count to zero, ARC eliminates idle runner costs during off-hours, weekends, and holidays. The tradeoff is cold-start latency: the first job after a scale-to-zero period must wait for a pod to schedule, pull its image, and register with GitHub. Pre-pulling runner images onto nodes and using GKE’s node auto-provisioning reduces this cold start to under 30 seconds in practice.

How It Works

ARC Controller Loop

The ARC controller runs as a Kubernetes deployment in the actions-runner-system namespace. It periodically polls the GitHub API to discover queued workflow runs, matches them against runner labels defined in RunnerDeployments, and adjusts replica counts. The reconciliation loop ensures that the desired state (enough runners for pending jobs) converges with the actual state (running pods).

Helm-Based Deployment

ARC is typically deployed using Helm charts. The infrastructure project’s repository defines values files per runner profile, making it straightforward to add new profiles or adjust resource limits.

# values-standard-runner.yaml
controllerServiceAccount:
  namespace: actions-runner-system
  name: arc-controller

githubConfigUrl: "https://github.com/myorg"
githubConfigSecret: github-app-secret

maxRunners: 10
minRunners: 0

containerMode:
  type: "kubernetes"

template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        resources:
          requests:
            cpu: "2"
            memory: "4Gi"
          limits:
            cpu: "2"
            memory: "4Gi"
    nodeSelector:
      cloud.google.com/gke-nodepool: ci-standard

Heavy Runner with Docker-in-Docker

For workflows that need to build container images, a Docker-in-Docker sidecar runs alongside the runner container, exposing the Docker socket via an emptyDir volume.

# values-docker-runner.yaml
githubConfigUrl: "https://github.com/myorg"
githubConfigSecret: github-app-secret

maxRunners: 5
minRunners: 0

template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        env:
          - name: DOCKER_HOST
            value: "unix:///var/run/docker.sock"
        volumeMounts:
          - name: docker-sock
            mountPath: /var/run/docker.sock
        resources:
          requests:
            cpu: "4"
            memory: "8Gi"
          limits:
            cpu: "8"
            memory: "16Gi"
      - name: dind
        image: docker:24-dind
        securityContext:
          privileged: true
        volumeMounts:
          - name: docker-sock
            mountPath: /var/run/docker.sock
          - name: dind-storage
            mountPath: /var/lib/docker
    volumes:
      - name: docker-sock
        emptyDir: {}
      - name: dind-storage
        emptyDir: {}
    nodeSelector:
      cloud.google.com/gke-nodepool: ci-heavy

ExternalSecrets for GitHub App Credentials

The GitHub App private key and metadata are stored in GCP Secret Manager and synced to Kubernetes via ExternalSecrets. This keeps sensitive credentials out of version control and Helm values.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: github-app-secret
  namespace: actions-runner-system
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-store
    kind: ClusterSecretStore
  target:
    name: github-app-secret
    template:
      type: Opaque
      data:
        github_app_id: "{{ .appId }}"
        github_app_installation_id: "{{ .installationId }}"
        github_app_private_key: "{{ .privateKey }}"
  data:
    - secretKey: appId
      remoteRef:
        key: arc-github-app-id
    - secretKey: installationId
      remoteRef:
        key: arc-github-app-installation-id
    - secretKey: privateKey
      remoteRef:
        key: arc-github-app-private-key

Practical Applications

A production infrastructure repository uses ARC to run all CI workflows for the platform. Standard runners handle Go linting (golangci-lint), unit tests, and documentation builds. Heavy runners execute integration test suites that spin up local database instances and run multi-node cluster tests. Docker runners build and push container images to Google Artifact Registry as part of the release pipeline.

By tagging runners with labels like self-hosted,linux,standard or self-hosted,linux,docker, workflow authors can target the right runner profile with a simple runs-on directive. This separation ensures that lightweight lint jobs don’t compete for resources with heavy compilation tasks, and that Docker builds have the privileged access they need without granting it to all runners.

Implementation Patterns

Graceful Scale-Down

ARC respects the terminationGracePeriodSeconds setting to allow running jobs to complete before a pod is terminated. Setting this to a value longer than your longest expected job (e.g., 3600 seconds) prevents mid-job termination during scale-down events. The controller also drains runners by marking them as “busy” in GitHub before removing the pod.

Ephemeral Runners

Configuring runners as ephemeral means each runner pod handles exactly one job, then self-terminates. This provides a clean environment for every workflow run — no state leakage between jobs — and simplifies the autoscaling model: the controller just needs to create one pod per pending job.

Runner Image Caching

Cold-start latency is dominated by container image pull time. Strategies to mitigate this include pre-pulling runner images onto node pools using a DaemonSet, using GKE Image Streaming for large images, and keeping runner images lean by installing only essential tooling and relying on workflow-level setup actions for language-specific tools.

Common Pitfalls

Further Reading