Structured Logging with Zap

Overview

Structured logging replaces free-form text log messages with machine-parseable records composed of typed key-value fields. Instead of writing log.Printf("user %s queried table %s in %dms", user, table, elapsed) and relying on regex to extract meaning later, structured logging produces a JSON (or similar) record like {"user":"alice","table":"events","elapsed_ms":42,"level":"info","msg":"query completed"}. Every field is named, typed, and independently queryable — log aggregation systems like Elasticsearch, Loki, or Datadog can index and filter on any field without parsing.

Zap is Uber’s high-performance structured logging library for Go. It achieves zero-allocation logging on the hot path by using a carefully designed API that avoids interface boxing and reflection. Zap provides two APIs: the SugaredLogger for convenience (printf-style with type inference) and the core Logger for performance-critical paths (strongly-typed fields, no allocations). In production Go services, Zap is commonly used for structured logging across both control-plane and data-plane components, with field-based log records that flow into centralized log aggregation for debugging, alerting, and compliance.

This topic covers the principles of structured logging, Zap’s architecture and API, log levels and scoped loggers, adapter patterns for library interoperability, and best practices for field naming and log hygiene.

Core Concepts

How It Works

A Zap logger is constructed from a zap.Config that specifies the log level, output encoding (JSON or console), output paths (stdout, stderr, files), and sampling rules. Once constructed, the logger is passed through the application via dependency injection or stored in a context. Each log call constructs a zapcore.Entry (timestamp, level, message, caller) and a slice of zapcore.Field values, which are encoded by the configured encoder and written to the configured output.

Creating a Production Logger

package main

import (
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

func NewLogger(level string) (*zap.Logger, error) {
    cfg := zap.Config{
        Level:       zap.NewAtomicLevelAt(parseLevel(level)),
        Development: false,
        Encoding:    "json",
        EncoderConfig: zapcore.EncoderConfig{
            TimeKey:        "ts",
            LevelKey:       "level",
            NameKey:        "logger",
            CallerKey:      "caller",
            MessageKey:     "msg",
            StacktraceKey:  "stacktrace",
            LineEnding:     zapcore.DefaultLineEnding,
            EncodeLevel:    zapcore.LowercaseLevelEncoder,
            EncodeTime:     zapcore.ISO8601TimeEncoder,
            EncodeDuration: zapcore.MillisDurationEncoder,
            EncodeCaller:   zapcore.ShortCallerEncoder,
        },
        OutputPaths:      []string{"stdout"},
        ErrorOutputPaths: []string{"stderr"},
    }
    return cfg.Build()
}

func parseLevel(s string) zapcore.Level {
    var lvl zapcore.Level
    _ = lvl.UnmarshalText([]byte(s))
    return lvl
}

Using Strongly-Typed Fields

The core Logger API uses typed field constructors to avoid allocations. Each field type has a dedicated constructor that encodes the value directly without boxing it into an interface{}.

func handleQuery(logger *zap.Logger, req *QueryRequest) {
    start := time.Now()

    // Create a scoped logger with request context.
    reqLogger := logger.With(
        zap.String("request_id", req.ID),
        zap.String("tenant", req.TenantID),
        zap.String("table", req.Table),
    )

    reqLogger.Info("query started",
        zap.String("query_type", req.Type),
        zap.Int("limit", req.Limit),
    )

    results, err := executeQuery(req)
    elapsed := time.Since(start)

    if err != nil {
        reqLogger.Error("query failed",
            zap.Error(err),
            zap.Duration("elapsed", elapsed),
        )
        return
    }

    reqLogger.Info("query completed",
        zap.Int("result_count", len(results)),
        zap.Duration("elapsed", elapsed),
        zap.Int64("bytes_scanned", results.BytesScanned),
    )
}

SugaredLogger for Convenience

The SugaredLogger wraps the core Logger with a more ergonomic API that accepts loosely-typed key-value pairs. It allocates on every call but is still faster than most logging libraries.

func processBackup(logger *zap.Logger, backupID string) {
    sugar := logger.Sugar()

    sugar.Infow("backup started",
        "backup_id", backupID,
        "strategy", "incremental",
    )

    // Printf-style for quick debugging (allocates).
    sugar.Debugf("scanning %d SSTables for backup %s", sstCount, backupID)

    sugar.Infow("backup completed",
        "backup_id", backupID,
        "duration_ms", elapsed.Milliseconds(),
        "size_bytes", totalBytes,
    )
}

Dynamic Log Level Changes

Zap’s AtomicLevel allows changing the log level at runtime via an HTTP endpoint — invaluable for debugging production issues without restarting the service.

func main() {
    atomicLevel := zap.NewAtomicLevelAt(zap.InfoLevel)

    logger, _ := zap.Config{
        Level:    atomicLevel,
        Encoding: "json",
        // ... other config
    }.Build()

    // Expose an HTTP endpoint to change log level at runtime.
    // PUT /log/level with body: {"level": "debug"}
    mux := http.NewServeMux()
    mux.Handle("/log/level", atomicLevel)

    go http.ListenAndServe(":9090", mux)

    // Now operators can enable debug logging on the fly:
    // curl -X PUT localhost:9090/log/level -d '{"level":"debug"}'
    logger.Info("server started", zap.String("log_level", atomicLevel.String()))
}

Adapter for gRPC Logging

When services communicate over gRPC, the gRPC library needs its own logger. Zap provides an adapter so gRPC logs flow through the same structured pipeline.

import (
    "go.uber.org/zap"
    "google.golang.org/grpc/grpclog"
)

func initGRPCLogging(logger *zap.Logger) {
    // Wrap the Zap logger to satisfy gRPC's LoggerV2 interface.
    grpcLogger := logger.Named("grpc").WithOptions(
        zap.IncreaseLevel(zap.WarnLevel), // Suppress gRPC's verbose INFO logs.
    )
    grpclog.SetLoggerV2(zapgrpc.NewLogger(grpcLogger))
}

Practical Applications

In a production Go service, Zap-based structured logging serves three primary purposes: operational debugging, alerting, and compliance audit trails. For example, a Kubernetes controller’s control plane uses scoped loggers per reconciliation loop — each reconciliation carries a logger enriched with the resource name, namespace, and generation number. When a reconciliation fails, operators can filter logs by {"logger":"controller.mydb","resource":"prod-cluster-1","generation":42} to see the complete sequence of events leading to the failure, without wading through logs from other resources.

An analytics service can use Zap with a Loki sink, where structured fields become Loki labels. Query performance logs include fields like tenant, table, query_type, and elapsed_ms, enabling dashboards that show p99 query latency broken down by tenant and table. Anomaly detection rules fire when elapsed_ms > 5000 for more than 5 consecutive queries from the same tenant.

Both services use Zap’s AtomicLevel to toggle debug logging in production. When an incident occurs, the on-call engineer bumps the affected pod to debug level via the /log/level endpoint, captures detailed logs for a few minutes, then restores info level — all without a restart or deployment.

Typical Integration Points

• HTTP middleware — log request method, path, status code, latency, and request ID for every request.
• gRPC interceptors — log RPC method, status code, and duration for every call via the Zap-gRPC adapter.
• Database operations — log query type, table, row count, and duration at the repository layer.
• Kubernetes controllers — scoped loggers per reconciliation with resource metadata fields.
• Background jobs — log job ID, type, and outcome with duration for post-mortem analysis.

Implementation Patterns

Logger Dependency Injection

Pass the logger as an explicit dependency rather than using a global. This makes testing easier (inject a zap.NewNop() or zaptest.NewLogger(t)) and ensures each component can add its own scoped fields without affecting other components.

type QueryEngine struct {
    logger *zap.Logger
    store  Storage
}

func NewQueryEngine(logger *zap.Logger, store Storage) *QueryEngine {
    return &QueryEngine{
        logger: logger.Named("query-engine"),
        store:  store,
    }
}

func (qe *QueryEngine) Execute(ctx context.Context, q Query) (Results, error) {
    qe.logger.Info("executing query",
        zap.String("query_id", q.ID),
        zap.String("table", q.Table),
    )
    // ...
}

Request-Scoped Logging with Context

Store the logger in the request context so that any function in the call chain can log with the request’s metadata without threading the logger through every function signature.

type ctxKey struct{}

func WithLogger(ctx context.Context, logger *zap.Logger) context.Context {
    return context.WithValue(ctx, ctxKey{}, logger)
}

func FromContext(ctx context.Context) *zap.Logger {
    if logger, ok := ctx.Value(ctxKey{}).(*zap.Logger); ok {
        return logger
    }
    return zap.NewNop()
}

// Middleware that injects a request-scoped logger.
func LoggingMiddleware(base *zap.Logger) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            reqLogger := base.With(
                zap.String("request_id", r.Header.Get("X-Request-ID")),
                zap.String("method", r.Method),
                zap.String("path", r.URL.Path),
            )
            ctx := WithLogger(r.Context(), reqLogger)
            next.ServeHTTP(w, r.WithContext(ctx))
        })
    }
}

Consistent Field Naming

Define field name constants to prevent typos and ensure consistency across the codebase. When every service uses "request_id" instead of a mix of "reqId", "req_id", and "requestID", log queries across services become reliable.

// pkg/logfields/fields.go
package logfields

const (
    RequestID  = "request_id"
    TenantID   = "tenant_id"
    UserID     = "user_id"
    Component  = "component"
    Table      = "table"
    ElapsedMS  = "elapsed_ms"
    BytesRead  = "bytes_read"
    ErrorCount = "error_count"
)

Common Pitfalls

Logging Sensitive Data

Structured logging makes it easy to add fields — too easy. Developers may inadvertently log passwords, API keys, authentication tokens, or PII (email addresses, IP addresses). Because structured logs are indexed and searchable, sensitive data in logs is more exposed than in unstructured text. Establish a deny-list of field names that must never appear in logs (password, token, secret, authorization) and enforce it with a custom Zap Core wrapper that redacts or drops matching fields.

Using SugaredLogger in Hot Paths

The SugaredLogger allocates on every call because it accepts interface{} arguments. In request handlers processing thousands of queries per second, these allocations compound into significant GC pressure. Use the core Logger with typed fields (zap.String, zap.Int, zap.Duration) in hot paths and reserve SugaredLogger for initialization code, CLI tools, and low-throughput paths where convenience outweighs performance.

Over-Logging at Info Level

Logging every database query, cache hit, and internal function call at Info level generates enormous log volumes that increase storage costs and make it harder to find important events. Use Debug for high-frequency operational details (individual query execution, cache lookups), Info for significant state changes (server started, configuration loaded, backup completed), Warn for recoverable anomalies (connection retry, slow query), and Error for failures that require attention (unhandled error, data corruption detected).

Forgetting to Sync the Logger

Zap buffers log output for performance. If the process crashes or exits without calling logger.Sync(), buffered log entries are lost — potentially including the error that caused the crash. Always defer logger.Sync() at the top of main(). Note that syncing stdout/stderr on some OS configurations returns a benign error; it’s safe to ignore the error from Sync() in a defer.

Unbounded Field Values

Logging large values — full request bodies, entire SQL queries, or base64-encoded blobs — as field values bloats log storage and can cause log aggregation systems to drop or truncate entries. Truncate large string fields to a reasonable maximum (e.g., 500 characters) and log large payloads at Debug level only. For request/response bodies, log a hash or size rather than the full content.

Further Reading