Building container images inside a Kubernetes cluster is a deceptively hard problem. The traditional approach — running docker build — requires a Docker daemon, which means either running Docker-in-Docker (security nightmare with privileged containers) or mounting the host’s Docker socket (exposes the host to container escapes). Kaniko solves this by building container images entirely in userspace: it executes each Dockerfile instruction by unpacking filesystem layers, applying changes, and snapshotting the result — all without a container runtime or elevated privileges.
The deploy platform takes this further by building a complete build controller around Kaniko. When a deployment is triggered, a background controller claims the pending build, fetches git credentials from Google Secret Manager, spawns a Kubernetes Job with an init container that clones the repository, then runs Kaniko to build and push the image. The controller streams build logs back to the user, handles failures gracefully, and writes the final image URI to the database so the deployment pipeline can proceed.
This architecture eliminates the need for an external CI system for container builds. The platform itself is the build system — users push code, and the platform builds, pushes, and deploys the image without the user configuring any CI pipelines. The entire flow is orchestrated by the build controller, with Kaniko providing the rootless, daemonless build capability that makes it safe to run inside a shared Kubernetes cluster.
Unlike docker build, which delegates to a Docker daemon that manages images, layers, and storage drivers, Kaniko operates as a single process that reads a Dockerfile and produces an OCI image. It extracts the base image into a build directory, executes RUN commands by forking processes (using the base image’s root filesystem), snapshots filesystem changes after each instruction, and pushes the final image to a registry. No daemon, no socket, no privileged containers — just a Go binary that understands Dockerfiles and OCI image manifests.
Kubernetes init containers run before the main container starts, sharing the same volumes. The build controller uses this pattern to separate concerns: an alpine/git init container clones the repository into a shared emptyDir volume, then Kaniko’s main container reads the cloned source from that volume. This separation keeps the Kaniko image minimal (no git binary needed), allows pinning specific git versions for reproducibility, and isolates git credential handling to the init container — the Kaniko container never sees the git token.
The build Job’s service account is bound to a GCP service account via Workload Identity, granting it permission to pull secrets from Google Secret Manager and push images to Artifact Registry. This eliminates static credentials: no long-lived tokens stored in Kubernetes Secrets, no credential rotation headaches. The GCP IAM binding is scoped to exactly the permissions the build Job needs — read secrets, push images — following the principle of least privilege.
Most frontend applications follow a predictable build pattern: install dependencies, run the build script, serve the output with a static file server. Rather than requiring every user to write a Dockerfile, the build controller generates a sensible default Dockerfile for recognized application types. If the repository contains a package.json with a build script and no Dockerfile, the controller generates a multi-stage Dockerfile that installs dependencies, builds the app, and copies the output to an nginx image. Users can override this by providing their own Dockerfile.
The controller follows the Kaniko Job’s pod logs in real time using the Kubernetes API’s log follow capability. Each log line is forwarded to the user’s deployment view (via WebSocket or SSE), providing the same build-log experience as a traditional CI system. If the build fails, the controller captures the error output and writes a structured error message to the deployment record, enabling users to diagnose failures without SSH-ing into the cluster.
The build controller runs as a long-lived goroutine that polls for pending deployments, claims one via an atomic database update, orchestrates the build, and updates the deployment with the result. The Kubernetes Job specification is constructed dynamically based on the deployment’s configuration.
package builder
import (
"context"
"fmt"
"log/slog"
"time"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
)
type Controller struct {
db *DB
k8s kubernetes.Interface
namespace string
registry string
logger *slog.Logger
}
func (c *Controller) Run(ctx context.Context) error {
ticker := time.NewTicker(5 * time.Second)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
deploy, err := c.db.ClaimPendingBuild(ctx)
if err != nil {
c.logger.Error("claiming build", "error", err)
continue
}
if deploy == nil {
continue // No pending builds.
}
go c.executeBuild(ctx, deploy)
}
}
}
func (c *Controller) executeBuild(ctx context.Context, deploy *Deployment) {
logger := c.logger.With("deployment_id", deploy.ID, "project", deploy.ProjectSlug)
logger.Info("starting build")
// Fetch git token from Secret Manager.
gitToken, err := c.fetchGitToken(ctx, deploy.ProjectID)
if err != nil {
c.db.FailBuild(ctx, deploy.ID, fmt.Sprintf("fetching git token: %v", err))
return
}
// Generate Dockerfile if not present in repo.
dockerfile := deploy.DockerfilePath
if dockerfile == "" {
dockerfile = "Dockerfile"
}
// Create and submit the Kubernetes Job.
job := c.buildJob(deploy, gitToken, dockerfile)
created, err := c.k8s.BatchV1().Jobs(c.namespace).Create(ctx, job, metav1.CreateOptions{})
if err != nil {
c.db.FailBuild(ctx, deploy.ID, fmt.Sprintf("creating build job: %v", err))
return
}
// Stream logs and wait for completion.
imageURI, err := c.waitForBuild(ctx, created.Name, deploy)
if err != nil {
c.db.FailBuild(ctx, deploy.ID, fmt.Sprintf("build failed: %v", err))
return
}
// Write the image URI back to the database.
c.db.CompleteBuild(ctx, deploy.ID, imageURI)
logger.Info("build completed", "image", imageURI)
}func (c *Controller) buildJob(deploy *Deployment, gitToken, dockerfile string) *batchv1.Job {
imageTag := fmt.Sprintf("%s/%s:%s",
c.registry, deploy.ProjectSlug, deploy.GitSHA[:8])
gitURL := fmt.Sprintf("https://x-access-token:%s@github.com/%s/%s.git",
gitToken, deploy.GitOrg, deploy.GitRepo)
backoffLimit := int32(0) // No retries — fail fast.
ttl := int32(3600) // Clean up completed Jobs after 1 hour.
return &batchv1.Job{
ObjectMeta: metav1.ObjectMeta{
Name: fmt.Sprintf("build-%s-%s", deploy.ProjectSlug, deploy.GitSHA[:8]),
Labels: map[string]string{
"app": "deploy-platform-builder",
"deployment-id": deploy.ID,
},
},
Spec: batchv1.JobSpec{
BackoffLimit: &backoffLimit,
TTLSecondsAfterFinished: &ttl,
Template: corev1.PodTemplateSpec{
Spec: corev1.PodSpec{
ServiceAccountName: "builder-sa",
RestartPolicy: corev1.RestartPolicyNever,
// Init container: clone the git repository.
InitContainers: []corev1.Container{
{
Name: "git-clone",
Image: "alpine/git:2.43.0",
Command: []string{"git", "clone",
"--depth=1",
"--branch", deploy.GitBranch,
gitURL,
"/workspace",
},
VolumeMounts: []corev1.VolumeMount{
{Name: "workspace", MountPath: "/workspace"},
},
},
},
// Main container: build with Kaniko.
Containers: []corev1.Container{
{
Name: "kaniko",
Image: "gcr.io/kaniko-project/executor:v1.22.0",
Args: []string{
fmt.Sprintf("--dockerfile=/workspace/%s", dockerfile),
"--context=/workspace",
fmt.Sprintf("--destination=%s", imageTag),
"--cache=true",
fmt.Sprintf("--cache-repo=%s/cache", c.registry),
"--snapshot-mode=redo",
"--log-format=text",
},
VolumeMounts: []corev1.VolumeMount{
{Name: "workspace", MountPath: "/workspace"},
},
},
},
Volumes: []corev1.Volume{
{
Name: "workspace",
VolumeSource: corev1.VolumeSource{
EmptyDir: &corev1.EmptyDirVolumeSource{},
},
},
},
},
},
},
}
}func generateDefaultDockerfile(appType string) string {
switch appType {
case "nextjs":
return `FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
FROM node:20-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public
EXPOSE 3000
CMD ["node", "server.js"]`
case "static":
return `FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
FROM nginx:1.25-alpine
COPY --from=builder /app/dist /usr/share/nginx/html
COPY --from=builder /app/nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]`
default:
return `FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --production
COPY . .
EXPOSE 3000
CMD ["npm", "start"]`
}
}The deploy platform uses this build controller to provide a “git push to deploy” experience. Users connect their GitHub repository, push code, and the platform automatically builds a container image and deploys it to Kubernetes. The build controller is the critical link between source code and running containers — it transforms a git commit into a deployable image without requiring users to write CI pipelines, manage Docker daemons, or understand container registries.
The init container pattern for git cloning has proven robust across thousands of builds. By pinning the alpine/git image to a specific version, builds are reproducible regardless of when they run. The shallow clone (--depth=1) keeps clone times under 10 seconds even for large repositories, and the ephemeral workspace volume ensures no source code persists after the build Job completes — important for security when building code from multiple tenants on a shared cluster.
Kaniko’s layer caching significantly reduces build times for iterative development. The first build of a Node.js application takes 2-3 minutes (downloading base images, installing dependencies), but subsequent builds with only source code changes complete in 30-60 seconds because the npm ci layer is cached. The cache is stored in the same container registry as the images, requiring no additional infrastructure.
Multiple controller replicas may run for high availability. To prevent duplicate builds, the controller uses an atomic database update to claim a pending build: UPDATE deployments SET status='building', builder_id=$1 WHERE status='pending' AND builder_id IS NULL LIMIT 1 RETURNING *. The WHERE builder_id IS NULL clause ensures exactly one controller claims each build, even under concurrent access.
Kaniko builds can hang if a RUN instruction enters an infinite loop or waits for user input. The controller sets an activeDeadlineSeconds on the Job spec (typically 10 minutes for standard builds, 30 minutes for large monorepos). When the deadline fires, Kubernetes kills the Job’s pods, and the controller marks the build as failed with a timeout message.
For platforms that need to support both amd64 and arm64 (e.g., deploying to mixed-architecture Kubernetes clusters), Kaniko can be run twice with different --custom-platform flags, producing separate images that are then combined into a manifest list. The build controller orchestrates this by spawning two parallel Jobs and combining the results.
Passing the git token as a command argument to the init container means it appears in the Job spec stored in etcd and in kubectl describe job output. In production, pass the token via a Kubernetes Secret mounted as an environment variable, and use git clone https://x-access-token:$GIT_TOKEN@github.com/... in a shell script rather than a direct git command. This prevents the token from appearing in the Job’s command-line arguments.
Kaniko’s cache key is based on the Dockerfile instruction text and the layer’s parent hash. If you change a build argument or environment variable that doesn’t appear in the Dockerfile instruction text, Kaniko may serve a stale cached layer. Use --cache-ttl=24h to set a maximum cache age, and ARG CACHE_BUST=$(date +%s) before instructions that must always re-execute.
If the alpine/git image can’t be pulled (registry outage, network partition), the entire build Job hangs in Init:ImagePullBackOff. Pre-pull init container images to cluster nodes using a DaemonSet, or mirror them to a private registry that’s closer to the cluster. Also set imagePullPolicy: IfNotPresent to use cached images when available.
The emptyDir volume uses the node’s ephemeral storage, which may be limited. Large repositories with binary assets can exhaust the node’s disk, causing the build and other pods on the same node to be evicted. Set sizeLimit on the emptyDir volume to fail the build early rather than destabilizing the node, and consider using a PersistentVolumeClaim for builds that need more space.
The default Dockerfile generator covers common frameworks but can’t handle every application type. When the generator doesn’t recognize the project structure, it falls back to a generic Node.js Dockerfile that may not work (e.g., the app uses Python or Go). Provide clear error messages when the default Dockerfile fails, and guide users to add their own Dockerfile with documentation links.