Datadog APM Across AWS, Redis, and Postgres

Overview

Distributed tracing connects the dots across a request’s journey through multiple services and infrastructure layers. When a user hits a SaaS API endpoint, the request might authenticate against a session store in Redis, query data from PostgreSQL, upload a file to S3 via the AWS SDK, and send an event to SQS — all within a single HTTP handler. Without distributed tracing, debugging a slow request requires correlating timestamps across separate log streams. With Datadog APM instrumentation, every operation becomes a span in a trace, and the entire request lifecycle is visible in a single flame graph.

A web application instruments its Go services using Datadog’s dd-trace-go library. Rather than manually creating spans throughout the codebase, the instrumentation wraps the infrastructure clients at initialization time: AWS sessions get wrapped with awstrace, Redis clients with redistrace, and sql.DB handles with sqltrace. Each wrapper automatically creates child spans for every operation — every S3 PutObject, every Redis GET, every SQL query — with the operation details (bucket name, Redis command, SQL query text) captured as span tags. Combined with expvar-based build info metrics, this gives the operations team both distributed traces for debugging individual requests and runtime metrics for capacity planning.

This topic covers the dd-trace instrumentation model, how to wrap each infrastructure client for automatic span creation, trace context propagation across service boundaries, span tag best practices, and the operational patterns for using traces effectively in production debugging.

Core Concepts

How It Works

Instrumentation follows a consistent pattern: initialize the tracer globally, wrap each infrastructure client at construction time, and let the wrappers handle span creation automatically. The tracer sends span data to the local Datadog agent (running as a sidecar or DaemonSet), which buffers, samples, and forwards traces to the Datadog backend.

Tracer Initialization

// internal/tracing/init.go
package tracing

import (
    "expvar"
    "os"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

var (
    buildCommit = expvar.NewString("build_commit")
    buildTime   = expvar.NewString("build_time")
    goVersion   = expvar.NewString("go_version")
)

// These are set at compile time via -ldflags.
var (
    Commit  string
    Built   string
    GoVer   string
)

func Init(serviceName string) {
    // Publish build info for the Datadog agent to scrape.
    buildCommit.Set(Commit)
    buildTime.Set(Built)
    goVersion.Set(GoVer)

    tracer.Start(
        tracer.WithService(serviceName),
        tracer.WithEnv(os.Getenv("DD_ENV")),              // "production", "staging"
        tracer.WithServiceVersion(Commit),
        tracer.WithRuntimeMetrics(),                        // goroutine count, GC stats
        tracer.WithAnalytics(true),                         // enable trace search
        tracer.WithAgentAddr(os.Getenv("DD_AGENT_HOST")+":8126"),
    )
}

func Shutdown() {
    tracer.Stop()
}

Wrapping AWS SDK Sessions

// internal/infra/aws.go
package infra

import (
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
    "github.com/aws/aws-sdk-go/service/sqs"
    awstrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/aws/aws-sdk-go/aws"
)

func NewTracedAWSSession(region string) *session.Session {
    sess := session.Must(session.NewSession(&aws.Config{
        Region: aws.String(region),
    }))

    // Wrap the session — every AWS API call now creates a span.
    // Span tags include: aws.service, aws.operation, aws.region,
    // and resource-specific tags (s3.bucket, sqs.queue_url).
    awstrace.WrapSession(sess)

    return sess
}

func NewS3Client(sess *session.Session) *s3.S3 {
    return s3.New(sess) // already traced via the wrapped session
}

func NewSQSClient(sess *session.Session) *sqs.SQS {
    return sqs.New(sess) // already traced via the wrapped session
}

Wrapping Redis with dd-trace

// internal/infra/redis.go
package infra

import (
    "context"

    "github.com/go-redis/redis/v8"
    redistrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/go-redis/redis.v8"
)

func NewTracedRedisClient(addr, password string, db int) *redis.Client {
    opts := &redis.Options{
        Addr:     addr,
        Password: password,
        DB:       db,
    }

    // redistrace.NewClient wraps the Redis client.
    // Every command (GET, SET, HGETALL, etc.) creates a span with:
    //   - redis.raw_command: the full command string
    //   - redis.args_length: number of arguments
    //   - out.host / out.port: Redis server address
    client := redistrace.NewClient(opts,
        redistrace.WithServiceName("redis"),
    )

    return client
}

// Usage — identical to standard go-redis, but every call is traced.
func ExampleUsage(ctx context.Context, client *redis.Client) {
    // This SET creates a span: service=redis, resource=SET, type=redis
    client.Set(ctx, "session:abc123", "user-data", 0)

    // This GET creates a span: service=redis, resource=GET, type=redis
    val := client.Get(ctx, "session:abc123")
    _ = val
}

Wrapping sql.DB with Traced Driver

// internal/infra/postgres.go
package infra

import (
    "database/sql"

    sqltrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/database/sql"
    "github.com/lib/pq"
)

func NewTracedPostgresDB(dsn string) (*sql.DB, error) {
    // Register the traced driver — wraps the standard pq driver.
    sqltrace.Register("postgres", &pq.Driver{},
        sqltrace.WithServiceName("postgres"),
        sqltrace.WithDBMPropagation(sqltrace.DBMPropagationModeService),
    )

    // Open using the traced driver name.
    db, err := sqltrace.Open("postgres", dsn)
    if err != nil {
        return nil, err
    }

    // Configure connection pool.
    db.SetMaxOpenConns(25)
    db.SetMaxIdleConns(10)

    return db, nil
}

// Every db.QueryContext, db.ExecContext, etc. now creates a span with:
//   - sql.query: the SQL text (with parameters normalized)
//   - db.type: "sql"
//   - db.instance: database name from DSN
//   - db.rows_affected: for write operations

HTTP Middleware for Trace Context

// internal/middleware/tracing.go
package middleware

import (
    "net/http"

    httptrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/net/http"
)

func TracingMiddleware(serviceName string) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return httptrace.WrapHandler(next, serviceName, "",
            httptrace.WithAnalytics(true),
        )
    }
}

// The middleware:
// 1. Extracts trace context from incoming request headers
// 2. Creates a root span for the HTTP handler
// 3. Injects the span into the request context
// 4. Child spans (Redis, Postgres, AWS) automatically attach
//    to the root span via context propagation
// 5. Sets span tags: http.method, http.url, http.status_code

Practical Applications

The web-app’s Datadog instrumentation enables three critical operational workflows. First, latency debugging: when a customer reports slow API responses, the on-call engineer searches Datadog for traces matching the customer’s tenant ID and time window. The flame graph immediately reveals whether the latency is in a PostgreSQL query (slow JOIN, missing index), a Redis call (cache miss leading to a round-trip), an AWS operation (S3 upload to a distant region), or application logic. Before instrumentation, this investigation required correlating timestamps across three different log systems.

Second, performance regression detection: Datadog’s trace analytics aggregate span durations across all requests. When a deployment increases the p99 latency of PostgreSQL queries from 15ms to 150ms, the regression appears in the service’s APM dashboard within minutes. The trace comparison view shows the exact query that regressed, and the deployment correlation (via the version tag set from the git commit) identifies which release caused it.

Third, dependency health monitoring: the service map — automatically generated from trace data — shows the web-app’s dependencies (Redis, PostgreSQL, S3, SQS) and their error rates. If the Redis cluster starts returning errors, the service map turns the Redis node red before any customer-facing impact is detected. The operations team can investigate Redis health before errors cascade into API failures.

Implementation Patterns

Centralized Client Construction

Wrap infrastructure clients in a single initialization function (or a dependencies struct) so that tracing is applied exactly once, at startup. Scattering client construction across handlers makes it easy to miss a wrapper, creating blind spots in the trace. In the web-app, all infrastructure clients are constructed in cmd/server/deps.go and injected into handlers via dependency injection.

// cmd/server/deps.go
type Dependencies struct {
    DB       *sql.DB
    Redis    *redis.Client
    S3       *s3.S3
    SQS      *sqs.SQS
    Logger   *zap.Logger
}

func NewDependencies(cfg Config) (*Dependencies, error) {
    awsSess := infra.NewTracedAWSSession(cfg.AWSRegion)

    db, err := infra.NewTracedPostgresDB(cfg.DatabaseURL)
    if err != nil {
        return nil, fmt.Errorf("postgres: %w", err)
    }

    return &Dependencies{
        DB:    db,
        Redis: infra.NewTracedRedisClient(cfg.RedisAddr, cfg.RedisPassword, 0),
        S3:    infra.NewS3Client(awsSess),
        SQS:   infra.NewSQSClient(awsSess),
    }, nil
}

Context Propagation Discipline

Every function that performs I/O must accept a context.Context and pass it to the infrastructure client. dd-trace uses the context to find the current parent span and attach child spans to it. If a handler calls db.Query(...) instead of db.QueryContext(ctx, ...), the SQL span is orphaned — it appears in Datadog as a separate trace, disconnected from the HTTP request. Enforce context usage with linter rules (e.g., contextcheck) and code review conventions.

Custom Span Enrichment

The automatic wrappers capture generic metadata, but business-specific context makes traces far more useful. Add custom tags to the root HTTP span for the tenant ID, user role, and feature flag values. These tags enable filtering traces by tenant (“show me all traces for tenant X”) and correlating performance with business dimensions (“are enterprise-tier tenants experiencing different latency than free-tier?”).

// Add business context to the current span.
func enrichSpan(ctx context.Context, tenantID, userRole string) {
    span, ok := tracer.SpanFromContext(ctx)
    if !ok {
        return
    }
    span.SetTag("tenant.id", tenantID)
    span.SetTag("user.role", userRole)
    span.SetTag("service.tier", lookupTier(tenantID))
}

Common Pitfalls

Missing Context Propagation

The single most common tracing bug: calling db.Query() instead of db.QueryContext(ctx). Without context, the traced driver has no parent span to attach to, and the SQL span either becomes a disconnected root or is dropped entirely. This also applies to Redis (client.Get(ctx, key) not client.Get(key)) and AWS SDK operations. Audit the codebase for I/O calls that don’t pass context — they’re tracing blind spots.

Sensitive Data in Span Tags

SQL query text in span tags may contain sensitive data: SELECT * FROM users WHERE email = 'user@example.com'. The dd-trace SQL wrapper normalizes parameter values by default (replacing literals with ?), but raw queries constructed with string interpolation bypass this normalization. Never build SQL queries with fmt.Sprintf — use parameterized queries, which both prevent SQL injection and ensure span tags don’t leak PII.

Trace Sampling Confusion

By default, the Datadog agent samples traces — not every trace is sent to the backend. This is necessary for cost control, but it means that searching for a specific trace by request ID might fail if that trace was sampled out. For debugging, use priority sampling: set manual.keep on spans for requests that match specific criteria (error responses, high-value tenants, latency above a threshold). This ensures critical traces are always retained.

Agent Connectivity Failures

The dd-trace library sends spans to a local Datadog agent (typically at localhost:8126). If the agent isn’t running — common in development, staging environments, or new Kubernetes pods before the DaemonSet is ready — span submission fails silently. The application continues to work, but no traces appear in Datadog. Add a startup health check that verifies agent connectivity, and log a warning (not an error) if the agent is unreachable so the failure is visible but non-blocking.

Over-Instrumentation and Cost

Tracing every function in the application creates massive traces with hundreds of spans, most of which are noise. Datadog charges by ingested span volume — over-instrumentation directly increases cost. Instrument at the I/O boundary: HTTP handlers, database calls, cache operations, and external API calls. Internal function calls are better served by CPU profiling (Datadog’s continuous profiler integrates with APM) than by tracing.

Further Reading