Kubernetes Operator Architecture for Databases

Overview

Running a distributed database like exampledb on Kubernetes requires more than simply deploying containers. Database nodes need stable network identities, persistent storage that survives pod restarts, ordered startup and shutdown sequences, and careful coordination between metadata nodes (which run the Raft consensus protocol) and data nodes (which store the actual vector and document data). A Kubernetes Operator encodes all of this operational knowledge into a custom controller that watches a CustomResource and continuously reconciles the desired state declared in the CR with the actual state of the cluster.

The database-operator defines an DatabaseCluster CustomResource that lets users declare the topology of their exampledb deployment — how many metadata nodes, how many data nodes, what storage class, what resource limits, whether to enable a service mesh sidecar, and cloud-specific tuning knobs for GKE Autopilot or EKS. The operator’s reconciliation loop reads this CR and creates or updates the corresponding Kubernetes objects: separate StatefulSets for metadata and data nodes, Services for internal cluster communication and external API access, ConfigMaps for node configuration, and PodDisruptionBudgets to protect quorum during voluntary disruptions.

This architecture follows the operator pattern popularized by CoreOS (now Red Hat) and formalized by the Operator SDK. The key insight is that a human operator’s runbook for managing a database cluster — scaling up, performing rolling upgrades, handling node failures, taking backups — can be encoded as a control loop that reacts to Kubernetes events and drives the cluster toward the desired state. The operator replaces manual kubectl commands and ad-hoc scripts with a declarative, version-controlled, and auditable resource definition.

Core Concepts

How It Works

The operator is built with the controller-runtime library (part of the Kubernetes controller ecosystem) and follows the standard manager-controller-reconciler architecture. A single manager process runs multiple controllers, each watching a different resource type.

DatabaseCluster CRD Spec

The CustomResource spec declares the full desired topology of the exampledb cluster. The operator reads this spec and creates the necessary Kubernetes objects.

# Example DatabaseCluster resource
apiVersion: example.com/v1alpha1
kind: DatabaseCluster
metadata:
  name: production
  namespace: database-system
spec:
  version: "1.8.2"
  metadata:
    replicas: 3
    resources:
      requests:
        cpu: "2"
        memory: "4Gi"
      limits:
        cpu: "4"
        memory: "8Gi"
    storage:
      size: 50Gi
      storageClassName: ssd-retain
  data:
    replicas: 5
    resources:
      requests:
        cpu: "4"
        memory: "16Gi"
      limits:
        cpu: "8"
        memory: "32Gi"
    storage:
      size: 500Gi
      storageClassName: ssd-retain
    spotPolicy:
      enabled: true
      gracefulShutdownSeconds: 120
  cloud:
    provider: gke
    autopilot: true
  serviceMesh:
    enabled: true
    provider: istio
  publicAPI:
    enabled: true
    type: LoadBalancer
    annotations:
      cloud.google.com/neg: '{"ingress": true}'

Reconciler Implementation

The core reconciler reads the DatabaseCluster CR, constructs the desired state for each owned object, and applies it using server-side apply. Owner references ensure that deleting the DatabaseCluster cascades to all owned objects.

// controllers/databasecluster_controller.go
package controllers

import (
    "context"
    "fmt"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"

    dbv1 "github.com/example/database-operator/api/v1alpha1"
)

type DatabaseClusterReconciler struct {
    client.Client
}

func (r *DatabaseClusterReconciler) Reconcile(
    ctx context.Context, req ctrl.Request,
) (ctrl.Result, error) {
    // Fetch the DatabaseCluster resource.
    var cluster dbv1.DatabaseCluster
    if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
        if errors.IsNotFound(err) {
            return ctrl.Result{}, nil // Deleted — owned objects cleaned up by GC.
        }
        return ctrl.Result{}, err
    }

    // Reconcile metadata StatefulSet.
    if err := r.reconcileMetadataStatefulSet(ctx, &cluster); err != nil {
        return ctrl.Result{}, fmt.Errorf("reconciling metadata sts: %w", err)
    }

    // Reconcile data StatefulSet.
    if err := r.reconcileDataStatefulSet(ctx, &cluster); err != nil {
        return ctrl.Result{}, fmt.Errorf("reconciling data sts: %w", err)
    }

    // Reconcile Services (internal + public API).
    if err := r.reconcileServices(ctx, &cluster); err != nil {
        return ctrl.Result{}, fmt.Errorf("reconciling services: %w", err)
    }

    // Update status with current replica counts.
    if err := r.updateStatus(ctx, &cluster); err != nil {
        return ctrl.Result{}, fmt.Errorf("updating status: %w", err)
    }

    // Requeue after 60s to recheck convergence.
    return ctrl.Result{RequeueAfter: 60 * time.Second}, nil
}

func (r *DatabaseClusterReconciler) reconcileMetadataStatefulSet(
    ctx context.Context, cluster *dbv1.DatabaseCluster,
) error {
    desired := r.buildMetadataStatefulSet(cluster)

    // Set owner reference for garbage collection.
    if err := controllerutil.SetControllerReference(cluster, desired, r.Scheme()); err != nil {
        return err
    }

    // Create or update using server-side apply semantics.
    existing := &appsv1.StatefulSet{}
    err := r.Get(ctx, client.ObjectKeyFromObject(desired), existing)
    if errors.IsNotFound(err) {
        return r.Create(ctx, desired)
    }
    if err != nil {
        return err
    }

    // Patch only fields we manage.
    existing.Spec.Replicas = desired.Spec.Replicas
    existing.Spec.Template = desired.Spec.Template
    return r.Update(ctx, existing)
}

func (r *DatabaseClusterReconciler) buildMetadataStatefulSet(
    cluster *dbv1.DatabaseCluster,
) *appsv1.StatefulSet {
    labels := map[string]string{
        "app.kubernetes.io/name":      "exampledb",
        "app.kubernetes.io/instance":  cluster.Name,
        "app.kubernetes.io/component": "metadata",
    }

    return &appsv1.StatefulSet{
        ObjectMeta: metav1.ObjectMeta{
            Name:      cluster.Name + "-metadata",
            Namespace: cluster.Namespace,
            Labels:    labels,
        },
        Spec: appsv1.StatefulSetSpec{
            Replicas:            &cluster.Spec.Metadata.Replicas,
            PodManagementPolicy: appsv1.OrderedReadyPodManagement,
            ServiceName:         cluster.Name + "-metadata-headless",
            Selector:            &metav1.LabelSelector{MatchLabels: labels},
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{Labels: labels},
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Name:      "exampledb-metadata",
                        Image:     fmt.Sprintf("exampledb/exampledb:%s", cluster.Spec.Version),
                        Args:      []string{"--role=metadata", "--data-dir=/data"},
                        Resources: cluster.Spec.Metadata.Resources,
                        Ports: []corev1.ContainerPort{
                            {Name: "raft", ContainerPort: 7000},
                            {Name: "grpc", ContainerPort: 7001},
                        },
                    }},
                },
            },
        },
    }
}

Setting Up the Controller Manager

The main.go entry point configures the manager, registers controllers, and starts the event loop. The manager handles leader election (ensuring only one operator instance is active), health checks, and graceful shutdown.

// main.go
package main

import (
    "os"

    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/healthz"

    dbv1 "github.com/example/database-operator/api/v1alpha1"
    "github.com/example/database-operator/controllers"
)

func main() {
    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        LeaderElection:   true,
        LeaderElectionID: "database-operator-lock",
        HealthProbeBindAddress: ":8081",
    })
    if err != nil {
        os.Exit(1)
    }

    // Register the DatabaseCluster CRD scheme.
    dbv1.AddToScheme(mgr.GetScheme())

    // Register the reconciler.
    if err := (&controllers.DatabaseClusterReconciler{
        Client: mgr.GetClient(),
    }).SetupWithManager(mgr); err != nil {
        os.Exit(1)
    }

    // Health and readiness probes.
    mgr.AddHealthzCheck("healthz", healthz.Ping)
    mgr.AddReadyzCheck("readyz", healthz.Ping)

    // Start the manager (blocks until shutdown signal).
    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        os.Exit(1)
    }
}

Practical Applications

The database-operator is the primary deployment mechanism for exampledb in production Kubernetes environments. Platform teams install the operator once (via Helm chart or OLM), then tenants create DatabaseCluster resources to provision isolated database instances. The operator handles day-2 operations that would otherwise require manual intervention: rolling version upgrades (metadata nodes first, then data nodes with shard draining), horizontal scaling (adding data nodes and triggering shard rebalancing), and recovery from node failures (detecting pods stuck in CrashLoopBackOff and replacing them with clean state).

The cloud-specific abstractions enable multi-cloud deployments from a single Helm chart. A customer running on GKE Autopilot sets cloud.provider: gke and cloud.autopilot: true, and the operator adjusts resource specifications, storage classes, and scheduling constraints accordingly. The same chart deployed on EKS with cloud.provider: eks configures IRSA annotations and EBS-optimized storage. This portability reduces the operational burden for the exampledb team, who maintain one operator codebase instead of separate deployment scripts per cloud.

The spot pod policy provides significant cost savings for data-heavy workloads. In benchmark deployments, running data nodes on spot instances reduced compute costs by 60-70%. The operator’s graceful shutdown handler ensures that when a spot instance is preempted, the affected data node drains its shards to surviving nodes before terminating, preventing data loss or query failures during the transition.

Typical Integration Points

• Helm charts and ArgoCD ApplicationSets that template DatabaseCluster resources for different environments (dev, staging, production) with appropriate sizing and cloud settings.
• Prometheus ServiceMonitors created by the operator to scrape exampledb metrics endpoints, enabling automated alerting on cluster health.
• PodDisruptionBudgets that protect Raft quorum during voluntary disruptions like node drains and Kubernetes upgrades.
• Istio VirtualServices and DestinationRules created when serviceMesh.enabled is true, providing mTLS between nodes and traffic management for the public API.

Implementation Patterns

Owner References and Garbage Collection

Every Kubernetes object created by the operator (StatefulSets, Services, ConfigMaps) has an owner reference pointing back to the DatabaseCluster resource. This enables Kubernetes garbage collection: when the DatabaseCluster is deleted, all owned objects are automatically cleaned up. The operator does not need explicit deletion logic — it relies on the platform’s built-in cascade semantics.

Status Conditions for Observability

The operator reports cluster health through standard Kubernetes conditions on the DatabaseCluster status: MetadataReady, DataReady, APIAvailable, and Degraded. Each condition includes a lastTransitionTime and a human-readable message. This enables external tools (kubectl, ArgoCD, custom dashboards) to monitor operator progress and detect issues without parsing logs.

Finalizers for Pre-Deletion Cleanup

The operator installs a finalizer on DatabaseCluster resources to perform cleanup that cannot be handled by garbage collection alone — such as deregistering the cluster from a central registry, draining all shards before tearing down nodes, or deleting external cloud resources (load balancers, DNS records) that Kubernetes does not manage. The finalizer logic runs before the resource is deleted and removes the finalizer upon completion, allowing the deletion to proceed.

Immutable Fields with Validation Webhooks

Certain fields in the DatabaseCluster spec should not be changed after creation — for example, the storage class and volume size of existing StatefulSets (Kubernetes does not support resizing PVCs in-place on all providers). The operator registers a validating admission webhook that rejects updates to immutable fields with a clear error message, preventing users from creating unreconcilable state.

Common Pitfalls

Reconciler Not Idempotent

The reconciler may be called multiple times for the same event, or called with stale state due to informer cache lag. If the reconciler creates a new object every time instead of checking whether one already exists, it produces duplicate StatefulSets or Services. The standard pattern is get-or-create: attempt to fetch the object by name, create it if not found, update it if it exists but differs from desired state.

Ignoring Raft Quorum During Scaling

Scaling metadata nodes (Raft voters) requires careful coordination. Adding a node means the new member must be registered with the Raft cluster before its pod starts. Removing a node means the member must be deregistered after its pod terminates. The operator cannot simply change the StatefulSet replica count — it must orchestrate Raft membership changes in lockstep with pod lifecycle events. Skipping this produces split-brain scenarios or clusters that cannot elect a leader.

Not Handling GKE Autopilot Resource Constraints

GKE Autopilot enforces that resource requests equal limits, does not support hostNetwork, and restricts certain security contexts. An operator that works on standard GKE will fail on Autopilot if it does not adjust its pod specs. The cloud.autopilot flag tells the operator to set requests == limits, remove unsupported fields, and use Autopilot-compatible node selectors. Omitting these adjustments produces pods stuck in Pending with cryptic Autopilot error messages.

StatefulSet Update Strategy Mismatch

StatefulSets support RollingUpdate and OnDelete update strategies. The operator uses OnDelete for metadata nodes to control the exact order of pod restarts during version upgrades (it deletes one pod at a time, waits for the replacement to become ready and rejoin the Raft group, then proceeds to the next). Using RollingUpdate for metadata nodes can cause Kubernetes to restart multiple voters simultaneously, violating quorum.

Leaking External Resources on Deletion

If the operator creates cloud load balancers, DNS records, or IAM bindings outside of Kubernetes, these resources are not covered by owner reference garbage collection. Without a finalizer that cleans up these external resources, deleting the DatabaseCluster leaves orphaned cloud resources that accumulate cost and security risk over time.

Further Reading