Metrics-Driven Data Node Autoscaler

Overview

Kubernetes provides the Horizontal Pod Autoscaler (HPA) for scaling workloads based on CPU and memory utilization. However, the HPA operates on simple threshold logic — if average CPU exceeds 70%, add a pod — which is too coarse for a distributed database. Database data nodes have correlated resource dimensions: high CPU usually coincides with high memory (due to query processing and caching), and scaling decisions must account for both dimensions simultaneously. Adding a node when CPU is high but memory is low may waste resources, while adding a node when memory is high but CPU is low may not help with the memory pressure because the new node starts empty and needs time to receive rebalanced shards.

A custom autoscaler controller can go beyond the HPA’s single-metric logic. It scrapes pod-level resource metrics from the Kubernetes Metrics API (provided by the metrics-server), computes per-node CPU and memory utilization as a percentage of requests, and only triggers a scale-up when both metrics exceed their respective thresholds. Scale-down follows the same dual-threshold logic in reverse: both metrics must be below their lower thresholds for a sustained cooldown period before the controller removes a replica. This “both must agree” policy prevents oscillation and wasted scaling events.

The controller also implements cooldown timers, minimum and maximum replica bounds, and integration with the shard rebalancer. After adding a node, the controller waits for the rebalancer to finish redistributing shards before evaluating metrics again — scaling based on metrics measured during rebalancing would produce incorrect decisions because the cluster is in a transient state.

Core Concepts

How It Works

The autoscaler runs as a separate controller within the database-operator manager process. It watches DatabaseCluster resources and runs an evaluation loop on a configurable interval (default: 30 seconds). Each evaluation fetches current metrics, applies the dual-threshold logic, checks cooldowns, and either patches the data StatefulSet’s replica count or does nothing.

Autoscaler Controller

// controllers/autoscaler_controller.go
package controllers

import (
    "context"
    "fmt"
    "time"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    metricsv1beta1 "k8s.io/metrics/pkg/apis/metrics/v1beta1"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"

    dbv1 "github.com/example/database-operator/api/v1alpha1"
)

type AutoscalerReconciler struct {
    client.Client
    MetricsClient client.Reader // reads from metrics.k8s.io
    lastScaleTime map[string]time.Time
}

type ScalingDecision int

const (
    ScaleNone ScalingDecision = iota
    ScaleUp
    ScaleDown
)

// evaluation holds the computed metrics for a scaling decision.
type evaluation struct {
    avgCPUPercent    float64
    avgMemoryPercent float64
    currentReplicas  int32
    decision         ScalingDecision
}

func (r *AutoscalerReconciler) Reconcile(
    ctx context.Context, req ctrl.Request,
) (ctrl.Result, error) {
    var cluster dbv1.DatabaseCluster
    if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Skip if autoscaling is disabled.
    if cluster.Spec.Data.Autoscaling == nil || !cluster.Spec.Data.Autoscaling.Enabled {
        return ctrl.Result{}, nil
    }

    // Check cooldown.
    key := req.NamespacedName.String()
    if last, ok := r.lastScaleTime[key]; ok {
        cooldown := time.Duration(cluster.Spec.Data.Autoscaling.CooldownSeconds) * time.Second
        if time.Since(last) < cooldown {
            return ctrl.Result{RequeueAfter: cooldown - time.Since(last)}, nil
        }
    }

    // Check if rebalancing is in progress.
    if r.isRebalancing(ctx, &cluster) {
        return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
    }

    // Evaluate metrics.
    eval, err := r.evaluate(ctx, &cluster)
    if err != nil {
        return ctrl.Result{}, fmt.Errorf("evaluating metrics: %w", err)
    }

    // Apply scaling decision.
    switch eval.decision {
    case ScaleUp:
        newReplicas := min(eval.currentReplicas+1, cluster.Spec.Data.Autoscaling.MaxReplicas)
        if newReplicas != eval.currentReplicas {
            if err := r.scaleDataNodes(ctx, &cluster, newReplicas); err != nil {
                return ctrl.Result{}, err
            }
            r.lastScaleTime[key] = time.Now()
        }
    case ScaleDown:
        newReplicas := max(eval.currentReplicas-1, cluster.Spec.Data.Autoscaling.MinReplicas)
        if newReplicas != eval.currentReplicas {
            if err := r.scaleDataNodes(ctx, &cluster, newReplicas); err != nil {
                return ctrl.Result{}, err
            }
            r.lastScaleTime[key] = time.Now()
        }
    }

    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

Metrics Evaluation Logic

The evaluation function fetches PodMetrics for all data node pods, computes average utilization, and applies the dual-threshold comparison.

// controllers/autoscaler_controller.go (continued)

func (r *AutoscalerReconciler) evaluate(
    ctx context.Context, cluster *dbv1.DatabaseCluster,
) (*evaluation, error) {
    as := cluster.Spec.Data.Autoscaling

    // List PodMetrics for data node pods.
    var podMetricsList metricsv1beta1.PodMetricsList
    if err := r.MetricsClient.List(ctx, &podMetricsList,
        client.InNamespace(cluster.Namespace),
        client.MatchingLabels{
            "app.kubernetes.io/instance":  cluster.Name,
            "app.kubernetes.io/component": "data",
        },
    ); err != nil {
        return nil, fmt.Errorf("listing pod metrics: %w", err)
    }

    if len(podMetricsList.Items) == 0 {
        return &evaluation{decision: ScaleNone}, nil
    }

    // Compute average CPU and memory utilization.
    var totalCPU, totalMemory float64
    for _, pm := range podMetricsList.Items {
        for _, c := range pm.Containers {
            if c.Name != "exampledb-data" {
                continue // Skip sidecar containers.
            }
            cpuMillis := float64(c.Usage.Cpu().MilliValue())
            memBytes := float64(c.Usage.Memory().Value())

            cpuRequest := float64(cluster.Spec.Data.Resources.Requests.Cpu().MilliValue())
            memRequest := float64(cluster.Spec.Data.Resources.Requests.Memory().Value())

            totalCPU += (cpuMillis / cpuRequest) * 100
            totalMemory += (memBytes / memRequest) * 100
        }
    }

    n := float64(len(podMetricsList.Items))
    avgCPU := totalCPU / n
    avgMem := totalMemory / n
    currentReplicas := int32(len(podMetricsList.Items))

    eval := &evaluation{
        avgCPUPercent:    avgCPU,
        avgMemoryPercent: avgMem,
        currentReplicas:  currentReplicas,
        decision:         ScaleNone,
    }

    // Dual-threshold: both must agree for a scaling action.
    if avgCPU > float64(as.CPUUpperThreshold) && avgMem > float64(as.MemoryUpperThreshold) {
        eval.decision = ScaleUp
    } else if avgCPU < float64(as.CPULowerThreshold) && avgMem < float64(as.MemoryLowerThreshold) {
        eval.decision = ScaleDown
    }

    return eval, nil
}

func (r *AutoscalerReconciler) scaleDataNodes(
    ctx context.Context, cluster *dbv1.DatabaseCluster, replicas int32,
) error {
    // Patch the data StatefulSet's replica count.
    stsName := cluster.Name + "-data"
    var sts appsv1.StatefulSet
    if err := r.Get(ctx, client.ObjectKey{
        Namespace: cluster.Namespace,
        Name:      stsName,
    }, &sts); err != nil {
        return fmt.Errorf("getting data statefulset: %w", err)
    }

    sts.Spec.Replicas = &replicas
    if err := r.Update(ctx, &sts); err != nil {
        return fmt.Errorf("scaling data statefulset to %d: %w", replicas, err)
    }

    // Record the scaling event on the cluster status.
    cluster.Status.LastScaleTime = &metav1.Time{Time: time.Now()}
    cluster.Status.DataReplicas = replicas
    return r.Status().Update(ctx, cluster)
}

Autoscaling Configuration in the CRD

# DatabaseCluster with autoscaling enabled
apiVersion: example.com/v1alpha1
kind: DatabaseCluster
metadata:
  name: production
spec:
  data:
    replicas: 5           # Initial replica count
    autoscaling:
      enabled: true
      minReplicas: 3
      maxReplicas: 20
      cpuUpperThreshold: 75     # Scale up when avg CPU > 75%
      cpuLowerThreshold: 30     # Scale down when avg CPU < 30%
      memoryUpperThreshold: 70  # Scale up when avg memory > 70%
      memoryLowerThreshold: 40  # Scale down when avg memory < 40%
      cooldownSeconds: 300      # 5 min between scaling events
      evaluationInterval: 30    # Check metrics every 30s

Practical Applications

The autoscaler addresses a fundamental operational challenge for distributed databases: database workloads are bursty. A product launch might 10x query traffic for a few hours, a batch import job might spike CPU for 30 minutes, and overnight traffic might drop to near zero. Without autoscaling, operators must either overprovision (wasting money 90% of the time) or risk degraded performance during spikes. The dual-threshold autoscaler balances cost and performance by scaling reactively to actual utilization while avoiding the pitfalls of single-metric scaling.

In production deployments, the autoscaler integrates with the shard rebalancer to ensure smooth scaling transitions. When a new node is added, the rebalancer moves shards from overloaded nodes to the new one, reducing per-node resource utilization organically. The autoscaler’s rebalance-awareness prevents it from interpreting the temporary rebalancing load as a signal to add yet more nodes, which would cascade into unnecessary over-provisioning.

The controller’s cooldown mechanism has proven essential in practice. Early versions without cooldowns exhibited “thrashing” during traffic ramps: a brief spike triggered a scale-up, the new node’s startup reduced average utilization below the lower threshold, triggering a scale-down, which increased utilization again, triggering another scale-up. The asymmetric cooldown (short for scale-up, long for scale-down) resolved this by giving the system time to stabilize after each change.

Typical Integration Points

• The Kubernetes Metrics API (metrics-server) providing real-time CPU and memory usage per pod — the autoscaler’s primary data source.
• The database cluster API’s rebalancing status endpoint, queried to detect active shard movements and suppress scaling during transitions.
• Prometheus alerting rules that fire when the autoscaler is unable to scale (e.g., already at maxReplicas but metrics are above the upper threshold).
• Operator status conditions that report autoscaler health, last scaling event time, and current evaluation metrics for dashboard visibility.

Implementation Patterns

Exponential Backoff on Scale-Up Failures

If a scale-up fails (e.g., insufficient cluster capacity, PVC provisioning timeout), the controller does not retry immediately on the next evaluation cycle. Instead, it applies exponential backoff to the requeue delay — 30s, 60s, 120s, up to 5 minutes — to avoid hammering the API server with doomed requests. The backoff resets on a successful scaling event or when the user manually adjusts the replica count.

Sliding Window Averaging

Rather than making scaling decisions on a single metrics snapshot (which may be noisy), the controller maintains a sliding window of the last 5 evaluations and computes a moving average. This smooths out transient spikes and brief dips that would otherwise trigger premature scaling. The window size is configurable, with larger windows providing more stability at the cost of slower reaction time.

Separate Scale-Up and Scale-Down Cooldowns

Scale-up should be responsive (users notice degraded performance quickly), while scale-down should be conservative (removing a node is more disruptive than adding one, because shards must be drained). The controller uses separate cooldown durations: a short scale-up cooldown (default: 3 minutes) and a longer scale-down cooldown (default: 10 minutes). This asymmetry reflects the operational reality that over-provisioning is cheaper than under-provisioning for a database.

Dry-Run Mode for Testing

The autoscaler supports a dryRun mode where it evaluates metrics and logs the scaling decision it would make, but does not actually change the replica count. This is invaluable during initial deployment: operators enable dry-run, observe the decisions in the operator logs, tune thresholds until the decisions match their expectations, then disable dry-run to activate real scaling. This prevents the autoscaler from making unexpected scaling decisions during the tuning phase.

Common Pitfalls

Using Limits Instead of Requests for Utilization

Computing utilization as usage / limits instead of usage / requests understates utilization when limits are higher than requests (the common configuration). A pod using 3 of 4 requested cores (75% utilization) but with a limit of 8 cores would appear as only 37.5% utilized. The autoscaler would not scale up, even though the pod is consuming its guaranteed resources. Always use requests as the denominator — they represent the resource level the pod is entitled to and the level at which the cluster capacity was planned.

Not Filtering Sidecar Containers

In service mesh deployments (Istio, Linkerd), each pod contains sidecar containers (Envoy proxy, init containers) whose resource usage should not factor into the autoscaler’s evaluation. The autoscaler must filter metrics to include only the exampledb-data container. Including sidecar metrics inflates the apparent utilization and triggers unnecessary scale-ups.

Scaling Down During Maintenance Windows

If the autoscaler removes a data node during a maintenance window (e.g., while a rolling upgrade is in progress), the upgrade controller and the autoscaler may fight over the replica count. The autoscaler should be either disabled during maintenance or aware of the cluster’s maintenance state. The database-operator solves this by checking for an example.com/maintenance annotation on the DatabaseCluster resource and suppressing scaling decisions when it is present.

Ignoring Pending Pods

If the cluster is at capacity and a scale-up creates a pod that stays in Pending (no schedulable node), the autoscaler should not attempt another scale-up on the next evaluation. The controller checks for pending data node pods and treats their existence as a “scale-up in progress” state, preventing duplicate scaling events. Without this check, the autoscaler can create multiple pending pods that waste cluster resources when they eventually schedule.

Oscillation from Asymmetric Thresholds That Are Too Close

If the upper threshold is 70% and the lower threshold is 65%, the system will oscillate: metrics hover around 67%, briefly crossing above 70% (triggering scale-up), which reduces per-node load to below 65% (triggering scale-down), which increases per-node load above 70% again. The recommended gap between upper and lower thresholds is at least 30 percentage points (e.g., upper 75%, lower 40%) to create a stable “dead zone” where no scaling action is taken.

Further Reading