Prometheus Metrics Implementation

Overview

Prometheus is the de facto standard for metrics collection in cloud-native systems. It uses a pull-based model where a Prometheus server scrapes HTTP endpoints exposed by applications, collecting time-series data that describes the system’s behavior over time. Unlike push-based systems where applications send metrics to a central collector, Prometheus’s pull model means applications only need to expose a /metrics endpoint in a specific text format — they don’t need to know where Prometheus is running or handle delivery failures.

In the exampledb ecosystem, Prometheus metrics are implemented in both Go (using the official prometheus/client_golang library) and Zig (using a custom metrics library that produces Prometheus-compatible exposition format). The Go services — including the database operator, the data node controller, and the API gateway — use counters, gauges, and histograms to track request rates, queue depths, latency distributions, and resource utilization. The Zig components — including the storage engine and the query executor — expose metrics through a lightweight HTTP handler that formats metric values into the Prometheus text exposition format.

This topic covers the three fundamental metric types (counters, gauges, histograms), bucket distribution design for histograms, the Prometheus text exposition format, and implementation patterns in both Go and Zig.

Core Concepts

How It Works

Implementing Prometheus metrics involves three steps: defining the metrics (choosing types, names, labels, and bucket distributions), instrumenting the code (recording observations at the right points), and exposing the metrics endpoint (serving the text exposition format over HTTP).

Go Implementation with client_golang

The Go services use the official Prometheus client library, which handles registration, collection, and exposition automatically.

// pkg/metrics/metrics.go
package metrics

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    // Counter: total number of requests, labeled by method and status.
    RequestsTotal = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Namespace: "exampledb",
            Subsystem: "api",
            Name:      "requests_total",
            Help:      "Total number of API requests.",
        },
        []string{"method", "status"},
    )

    // Gauge: number of currently active queries.
    ActiveQueries = promauto.NewGauge(prometheus.GaugeOpts{
        Namespace: "exampledb",
        Subsystem: "query",
        Name:      "active_queries",
        Help:      "Number of queries currently being executed.",
    })

    // Histogram: request latency distribution.
    RequestDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Namespace: "exampledb",
            Subsystem: "api",
            Name:      "request_duration_seconds",
            Help:      "Request latency in seconds.",
            Buckets:   []float64{0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0},
        },
        []string{"method", "endpoint"},
    )
)

Instrumenting HTTP Handlers in Go

Metrics are recorded at the points where the measured events occur. For HTTP handlers, this means incrementing the request counter and observing the duration at the end of each request.

// pkg/middleware/metrics.go
package middleware

import (
    "net/http"
    "strconv"
    "time"

    "exampledb/pkg/metrics"
)

func MetricsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()

        // Wrap ResponseWriter to capture status code.
        rw := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}

        // Track active queries (gauge).
        metrics.ActiveQueries.Inc()
        defer metrics.ActiveQueries.Dec()

        next.ServeHTTP(rw, r)

        duration := time.Since(start).Seconds()
        status := strconv.Itoa(rw.statusCode)

        // Record counter and histogram.
        metrics.RequestsTotal.WithLabelValues(r.Method, status).Inc()
        metrics.RequestDuration.WithLabelValues(r.Method, r.URL.Path).Observe(duration)
    })
}

type responseWriter struct {
    http.ResponseWriter
    statusCode int
}

func (rw *responseWriter) WriteHeader(code int) {
    rw.statusCode = code
    rw.ResponseWriter.WriteHeader(code)
}

Exposing the /metrics Endpoint in Go

The Prometheus client library provides a handler that serializes all registered metrics into the text exposition format.

// cmd/server/main.go
package main

import (
    "net/http"

    "github.com/prometheus/client_golang/prometheus/promhttp"

    "exampledb/pkg/middleware"
)

func main() {
    mux := http.NewServeMux()

    // Application routes wrapped with metrics middleware.
    mux.Handle("/api/", middleware.MetricsMiddleware(apiHandler()))

    // Prometheus metrics endpoint (no middleware to avoid self-instrumentation).
    mux.Handle("/metrics", promhttp.Handler())

    http.ListenAndServe(":8080", mux)
}

Zig Implementation: Custom Prometheus Exposition

The Zig storage engine implements Prometheus metrics without an external library, directly producing the text exposition format. This avoids C library dependencies and keeps the binary size minimal.

// src/metrics.zig
const std = @import("std");

pub const MetricType = enum {
    counter,
    gauge,
    histogram,
};

pub const Counter = struct {
    value: std.atomic.Value(u64) = std.atomic.Value(u64).init(0),

    pub fn inc(self: *Counter) void {
        _ = self.value.fetchAdd(1, .monotonic);
    }

    pub fn add(self: *Counter, n: u64) void {
        _ = self.value.fetchAdd(n, .monotonic);
    }

    pub fn get(self: *const Counter) u64 {
        return self.value.load(.monotonic);
    }
};

pub const Gauge = struct {
    value: std.atomic.Value(i64) = std.atomic.Value(i64).init(0),

    pub fn set(self: *Gauge, val: i64) void {
        self.value.store(val, .monotonic);
    }

    pub fn inc(self: *Gauge) void {
        _ = self.value.fetchAdd(1, .monotonic);
    }

    pub fn dec(self: *Gauge) void {
        _ = self.value.fetchSub(1, .monotonic);
    }

    pub fn get(self: *const Gauge) i64 {
        return self.value.load(.monotonic);
    }
};

pub const Histogram = struct {
    bounds: []const f64,
    buckets: []std.atomic.Value(u64),
    sum: std.atomic.Value(u64) = std.atomic.Value(u64).init(0),
    count: std.atomic.Value(u64) = std.atomic.Value(u64).init(0),
    allocator: std.mem.Allocator,

    pub fn init(allocator: std.mem.Allocator, bounds: []const f64) !Histogram {
        const buckets = try allocator.alloc(std.atomic.Value(u64), bounds.len + 1);
        for (buckets) |*b| b.* = std.atomic.Value(u64).init(0);
        return .{
            .bounds = bounds,
            .buckets = buckets,
            .allocator = allocator,
        };
    }

    pub fn observe(self: *Histogram, value: f64) void {
        // Increment the appropriate bucket(s) — cumulative.
        for (self.bounds, 0..) |bound, i| {
            if (value <= bound) {
                _ = self.buckets[i].fetchAdd(1, .monotonic);
            }
        }
        // +Inf bucket always gets incremented.
        _ = self.buckets[self.bounds.len].fetchAdd(1, .monotonic);

        // Update sum (store as fixed-point u64: value * 1_000_000).
        const fixed = @as(u64, @intFromFloat(value * 1_000_000));
        _ = self.sum.fetchAdd(fixed, .monotonic);
        _ = self.count.fetchAdd(1, .monotonic);
    }
};

Zig: Formatting the Exposition Output

// src/metrics_handler.zig
const std = @import("std");
const metrics = @import("metrics.zig");

pub fn writeMetrics(writer: anytype, registry: *const Registry) !void {
    // Counter example.
    try writer.print("# HELP exampledb_storage_reads_total Total read operations.\n", .{});
    try writer.print("# TYPE exampledb_storage_reads_total counter\n", .{});
    try writer.print("exampledb_storage_reads_total {d}\n", .{registry.reads_total.get()});

    // Gauge example.
    try writer.print("# HELP exampledb_storage_memtable_bytes Current memtable size in bytes.\n", .{});
    try writer.print("# TYPE exampledb_storage_memtable_bytes gauge\n", .{});
    try writer.print("exampledb_storage_memtable_bytes {d}\n", .{registry.memtable_bytes.get()});

    // Histogram example.
    try writer.print("# HELP exampledb_storage_read_duration_seconds Read latency.\n", .{});
    try writer.print("# TYPE exampledb_storage_read_duration_seconds histogram\n", .{});
    const h = ®istry.read_duration;
    for (h.bounds, 0..) |bound, i| {
        try writer.print("exampledb_storage_read_duration_seconds_bucket{{le=\"{d:.6}\"}} {d}\n", .{ bound, h.buckets[i].load(.monotonic) });
    }
    try writer.print("exampledb_storage_read_duration_seconds_bucket{{le=\"+Inf\"}} {d}\n", .{h.buckets[h.bounds.len].load(.monotonic)});
    try writer.print("exampledb_storage_read_duration_seconds_sum {d:.6}\n", .{@as(f64, @floatFromInt(h.sum.load(.monotonic))) / 1_000_000.0});
    try writer.print("exampledb_storage_read_duration_seconds_count {d}\n", .{h.count.load(.monotonic)});
}

Practical Applications

In the exampledb ecosystem, Prometheus metrics drive both operational dashboards and automated scaling decisions. The Metrics-Driven Data Node Autoscaler (covered in a separate topic) reads exampledb_storage_memtable_bytes and exampledb_query_active_queries gauges to determine when to scale up or down. Grafana dashboards visualize request rates (rate(exampledb_api_requests_total[5m])), p99 latencies (histogram_quantile(0.99, rate(exampledb_api_request_duration_seconds_bucket[5m]))), and error rates (rate(exampledb_api_requests_total{status=~"5.."}[5m])).

The Go operator uses counters to track reconciliation loops (exampledb_operator_reconciliations_total), with labels distinguishing successful reconciliations from errors. This enables alerting on reconciliation failure rates — if rate(exampledb_operator_reconciliations_total{result="error"}[10m]) > 0.1, the oncall team is paged because the operator is failing to converge.

The Zig storage engine exposes histograms for read and write latencies with carefully chosen bucket boundaries that align with the system’s SLO targets. The buckets [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5] (100us to 500ms) provide good resolution around the p99 SLO of 10ms, with enough headroom to observe tail latencies during compaction storms.

Typical Integration Points

• API gateway — request rate, latency distribution, and error rate by endpoint and method.
• Storage engine — read/write latency, memtable size, SSTable count, compaction duration.
• Query executor — active queries gauge, query parse time, execution time histograms.
• Kubernetes operator — reconciliation rate, reconciliation duration, error rate by resource type.
• Backup system — backup duration, backup size, last successful backup timestamp gauge.

Implementation Patterns

Histogram Bucket Design

Bucket boundaries should align with SLO thresholds and expected value distributions. For HTTP latencies, use exponential buckets that provide resolution where it matters (around p50 and p99 targets) and coarser buckets for tail latencies. The prometheus.ExponentialBuckets(0.001, 2, 12) helper generates [0.001, 0.002, 0.004, ..., 2.048], which covers 1ms to 2s with doubling intervals. Avoid more than 20 buckets per histogram — each bucket creates a separate time series, and high bucket counts multiply cardinality.

Naming Conventions

Follow the Prometheus naming best practices: use namespace_subsystem_name_unit format. Counters end with _total, histograms measuring seconds end with _seconds, and bytes metrics end with _bytes. Use snake_case exclusively. The namespace should be the application name (e.g., exampledb), and the subsystem should identify the component (api, storage, query).

// Good names:
// exampledb_api_requests_total
// exampledb_storage_read_duration_seconds
// exampledb_query_active_queries
// exampledb_storage_memtable_bytes

// Bad names:
// requests (no namespace, no _total)
// api_latency (no unit suffix, no namespace)
// exampledb_numberOfActiveQueries (camelCase)

Avoiding High Cardinality

Labels with unbounded values — user IDs, request IDs, full URLs with query parameters — create cardinality explosions. Use bounded labels: HTTP methods (GET, POST, PUT, DELETE), status code classes (2xx, 4xx, 5xx), route patterns (/api/v1/users/:id), or component names. If you need per-user metrics, emit them as logs or traces, not Prometheus metrics.

// Good: bounded labels
metrics.RequestsTotal.WithLabelValues("GET", "200").Inc()
metrics.RequestsTotal.WithLabelValues("POST", "500").Inc()

// Bad: unbounded label values — creates millions of time series
metrics.RequestsTotal.WithLabelValues("GET", "200", userID).Inc()
metrics.RequestsTotal.WithLabelValues("GET", "200", requestURL).Inc()

Common Pitfalls

Counter Resets and rate() Windows

When a process restarts, counters reset to zero. Prometheus handles this with reset detection in rate() — it detects when a counter decreases and compensates. However, if the rate() window is shorter than two scrape intervals, the reset cannot be detected, producing incorrect spikes. Always use a rate() window of at least 4 * scrape_interval (e.g., [2m] for a 30s scrape interval) to ensure accurate results across restarts.

Histogram Bucket Boundaries After Deployment

Changing histogram bucket boundaries after deployment creates discontinuities in time-series data. Prometheus stores each bucket as a separate time series, so adding or removing buckets creates new series and orphans old ones. histogram_quantile() queries that span the boundary change produce incorrect results. If bucket boundaries must change, deploy the change with a new metric name (versioned suffix like _v2) and migrate dashboards and alerts to the new metric.

Gauge Anti-patterns: Using Gauges for Counters

A common mistake is using a gauge to track something that should be a counter — for example, setting a gauge to the total number of processed items. If the process restarts, the gauge resets to zero and stays there until items are processed again. Counters handle resets gracefully via rate(); gauges do not. Use counters for cumulative totals and gauges only for values that naturally fluctuate (queue depth, memory usage, temperature).

Self-Instrumenting the /metrics Endpoint

If the metrics middleware wraps the /metrics endpoint itself, every Prometheus scrape generates additional metric observations. With a 15-second scrape interval, this adds 5,760 observations per day just from scraping. Exclude the metrics endpoint from instrumentation middleware to keep metrics clean and avoid circular feedback loops.

Zig Atomic Ordering Choices

The Zig metrics implementation uses .monotonic atomic ordering for counter increments, which provides correctness for single-value updates but does not guarantee visibility ordering across multiple metrics. If two metrics must be consistent with each other (e.g., bytes_written and writes_total), a missed observation on one but not the other during a Prometheus scrape can produce temporary inconsistencies. For metrics that must be scraped atomically, use a mutex-protected snapshot rather than individual atomic reads.

Further Reading