Docker Compose Multi-Service Orchestration

Overview

Modern applications rarely run as a single process. A typical service depends on a database, a cache layer, a message broker, and often several other microservices. During development, each developer needs a reproducible way to spin up this entire constellation of services with correct networking, volumes, and environment variables. Docker Compose solves this by letting you declare your multi-service architecture in a single YAML file and bring it all up with one command.

In a typical search application, Docker Compose orchestrates the service alongside its dependencies — including database services, indexing workers, and frontend servers. The docker-compose.yml file defines service dependencies, startup ordering, shared networks, volume mounts for persistent data, and environment configuration that would otherwise require a multi-page setup guide.

Understanding Docker Compose orchestration is essential for any developer working on distributed systems. It bridges the gap between “it works on my machine” and consistent, reproducible development environments that mirror production topology without the overhead of a full Kubernetes cluster.

Core Concepts

How It Works

A Docker Compose file describes the desired state of a multi-service application. When you run docker compose up, the Compose engine reads the YAML, resolves dependencies, creates networks and volumes, pulls or builds images, and starts containers in the correct order.

Multi-Service Application Definition

# docker-compose.yml — search application development environment
version: "3.9"

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: myapp
      POSTGRES_PASSWORD: ${DB_PASSWORD:-devpassword}
    ports:
      - "5432:5432"
    volumes:
      - pg_data:/var/lib/postgresql/data
      - ./migrations/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myapp"]
      interval: 5s
      timeout: 3s
      retries: 5

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  indexer:
    build:
      context: .
      dockerfile: Dockerfile.indexer
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      DATABASE_URL: postgres://myapp:${DB_PASSWORD:-devpassword}@postgres:5432/myapp
      REDIS_URL: redis://redis:6379
      INDEX_BATCH_SIZE: ${INDEX_BATCH_SIZE:-1000}
    volumes:
      - ./data:/app/data:ro

  api:
    build:
      context: .
      dockerfile: Dockerfile.api
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      indexer:
        condition: service_started
    ports:
      - "8080:8080"
    environment:
      DATABASE_URL: postgres://myapp:${DB_PASSWORD:-devpassword}@postgres:5432/myapp
      REDIS_URL: redis://redis:6379
      LOG_LEVEL: ${LOG_LEVEL:-info}
    volumes:
      - ./src:/app/src:ro

volumes:
  pg_data:
  redis_data:

Healthcheck-Gated Startup

The healthcheck definitions above are critical for reliable startup. Without them, depends_on only waits for the container to start — not for the process inside to be ready. A Go service that connects to PostgreSQL on startup will fail if Postgres is still initializing. The service_healthy condition solves this by waiting until pg_isready returns success.

# Start all services with dependency-ordered, healthcheck-gated startup
docker compose up -d

# View real-time logs from all services
docker compose logs -f

# Restart only the API service (dependencies stay running)
docker compose restart api

# Rebuild and restart the indexer after code changes
docker compose up -d --build indexer

# Tear down containers but preserve volumes
docker compose down

# Tear down containers AND delete volumes (destructive)
docker compose down -v

Override Files for Environment Variation

# docker-compose.override.yml — Development-specific overrides
# Compose automatically merges this with docker-compose.yml
version: "3.9"

services:
  api:
    build:
      target: development
    volumes:
      - ./src:/app/src  # Live reload via bind mount
    environment:
      LOG_LEVEL: debug
      HOT_RELOAD: "true"
    ports:
      - "9229:9229"  # Debugger port

# docker-compose.ci.yml — CI-specific configuration
# Used with: docker compose -f docker-compose.yml -f docker-compose.ci.yml up
version: "3.9"

services:
  api:
    build:
      target: production
    environment:
      LOG_LEVEL: warn
      CI: "true"

  test-runner:
    build:
      context: .
      dockerfile: Dockerfile.test
    depends_on:
      api:
        condition: service_healthy
    environment:
      API_URL: http://api:8080

Practical Applications

In a typical search application, Docker Compose defines the complete development environment: a PostgreSQL database for metadata storage, a Redis instance for caching and job queues, an indexer service that processes documents into searchable indices, and an API server that handles search queries. A new developer clones the repo and runs docker compose up to get a fully functional search system without installing any dependencies locally beyond Docker.

The same Compose configuration serves CI pipelines. GitHub Actions runs docker compose -f docker-compose.yml -f docker-compose.ci.yml up -d to spin up the full stack, then executes integration tests against the API service. When tests complete, docker compose down -v cleans up all containers and volumes, leaving the runner in a pristine state for the next build.

For load testing, the Compose file can be extended with a docker-compose.loadtest.yml override that scales the indexer service to multiple replicas (docker compose up -d --scale indexer=4) and adds monitoring services like Prometheus and Grafana to observe system behavior under load.

Implementation Patterns

Environment File Layering

Use a .env file for secrets and machine-specific configuration, while keeping defaults in the Compose file itself via ${VAR:-default} syntax. This way the Compose file is self-documenting (you can see all configuration knobs and their defaults) while sensitive values like database passwords stay out of version control.

Init Containers via depends_on

Short-lived setup tasks (database migrations, seed data loading) can be modeled as services with restart: "no". Downstream services depend on them with condition: service_completed_successfully, ensuring migrations run exactly once before the application starts.

services:
  migrate:
    build: .
    command: ["./migrate", "up"]
    depends_on:
      postgres:
        condition: service_healthy
    restart: "no"

  api:
    depends_on:
      migrate:
        condition: service_completed_successfully

Profiles for Optional Services

Use Compose profiles to group optional services that aren’t always needed. Monitoring tools, debug utilities, and load generators can be assigned to profiles and only started when explicitly requested.

services:
  grafana:
    image: grafana/grafana:latest
    profiles: ["monitoring"]
    ports:
      - "3000:3000"

# Start with monitoring: docker compose --profile monitoring up -d
# Start without: docker compose up -d (grafana won't start)

Common Pitfalls

Relying on depends_on Without Healthchecks

The default depends_on only waits for the container to be created, not for the service inside to be ready. Applications that connect to databases or APIs on startup will fail intermittently. Always pair depends_on with healthchecks and condition: service_healthy for services that need time to initialize.

Bind Mount Performance on macOS

Docker Desktop on macOS uses a Linux VM, and bind mounts between the macOS host and the VM have significant I/O overhead. For projects with many small files (like node_modules), this can make builds and file watches painfully slow. Use named volumes for directories that don’t need host access, and use :cached or :delegated consistency flags for bind mounts that do.

Port Conflicts

If another process on the host is already listening on a mapped port (e.g., a locally installed PostgreSQL on 5432), the container will fail to start. Use non-standard host ports ("15432:5432") or configure port mappings through environment variables so each developer can avoid conflicts with their local setup.

Orphaned Volumes Consuming Disk

Named volumes persist even after docker compose down. Over time, volumes from old projects accumulate and consume significant disk space. Run docker volume prune periodically, but be aware it removes all unused volumes — not just those from the current project.

Further Reading