Agent Runtime Sync Automation

Overview

Multi-agent systems that run as long-lived services face a configuration synchronization challenge: agent definitions, routing configurations, and API credentials must be consistent across the repository’s source of truth, the runtime directories where agents execute, and the gateway service that routes requests to them. Manual synchronization is error-prone — a developer updates an agent’s prompt in the repository but forgets to copy it to the runtime directory, leading to stale behavior in production.

This pattern is typically implemented with a pair of shell scripts — setup.sh and sync-runtime.sh — that automate the entire synchronization lifecycle. These scripts load secrets from .env files, verify that required API tokens (Anthropic, Discord) are valid, copy canonical agent definitions and gateway configuration into both the repository’s config/ directory and mounted runtime directories, seed per-agent workspace directories, and optionally restart the gateway after syncing to pick up changes.

This pattern of “canonical source + automated sync + validation gates” is a practical alternative to full configuration management systems like Ansible or Puppet for smaller deployments. It provides the safety guarantees that matter most — secrets are verified before use, configuration is consistent across all locations, and restarts happen only when something changed — without the learning curve or operational overhead of enterprise tooling.

Core Concepts

How It Works

The automation consists of two scripts with distinct responsibilities: setup.sh handles first-time environment preparation, and sync-runtime.sh handles ongoing configuration synchronization.

Environment Setup Script

#!/usr/bin/env bash
# setup.sh — First-time environment setup for multi-agent system.
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${SCRIPT_DIR}/.env"

# --- Load secrets ---
if [[ ! -f "$ENV_FILE" ]]; then
    echo "ERROR: .env file not found at $ENV_FILE"
    echo "Copy .env.example to .env and fill in your API keys."
    exit 1
fi
source "$ENV_FILE"

# --- Validate required secrets ---
validate_anthropic_key() {
    local response
    response=$(curl -s -o /dev/null -w "%{http_code}" \
        -H "x-api-key: ${ANTHROPIC_API_KEY}" \
        -H "anthropic-version: 2023-06-01" \
        "https://api.anthropic.com/v1/models")
    if [[ "$response" != "200" ]]; then
        echo "ERROR: Anthropic API key validation failed (HTTP $response)"
        exit 1
    fi
    echo "✓ Anthropic API key is valid"
}

validate_discord_token() {
    local response
    response=$(curl -s -o /dev/null -w "%{http_code}" \
        -H "Authorization: Bot ${DISCORD_BOT_TOKEN}" \
        "https://discord.com/api/v10/users/@me")
    if [[ "$response" != "200" ]]; then
        echo "ERROR: Discord bot token validation failed (HTTP $response)"
        exit 1
    fi
    echo "✓ Discord bot token is valid"
}

echo "Validating API credentials..."
validate_anthropic_key
validate_discord_token

# --- Create per-agent workspaces ---
AGENTS_DIR="${SCRIPT_DIR}/agents"
RUNTIME_DIR="${RUNTIME_BASE:-/var/lib/agent-runtime}"

for agent_dir in "$AGENTS_DIR"/*/; do
    agent_name=$(basename "$agent_dir")
    workspace="${RUNTIME_DIR}/workspaces/${agent_name}"

    echo "Seeding workspace for agent: $agent_name"
    mkdir -p "$workspace"/{data,logs,tmp}

    # Copy initial state if the agent defines one.
    if [[ -f "${agent_dir}/initial-state.json" ]]; then
        cp -n "${agent_dir}/initial-state.json" "${workspace}/state.json"
    fi
done

echo "Setup complete. Run sync-runtime.sh to deploy configuration."

Runtime Sync Script

#!/usr/bin/env bash
# sync-runtime.sh — Sync canonical agent configs to runtime directories.
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "${SCRIPT_DIR}/.env"

AGENTS_DIR="${SCRIPT_DIR}/agents"
CONFIG_DIR="${SCRIPT_DIR}/config"
RUNTIME_DIR="${RUNTIME_BASE:-/var/lib/agent-runtime}"
GATEWAY_CONFIG="${SCRIPT_DIR}/gateway.json"

# Track whether anything changed.
CHANGES=0

# --- Sync agent definitions ---
sync_file() {
    local src="$1" dst="$2"
    if [[ ! -f "$dst" ]] || ! cmp -s "$src" "$dst"; then
        cp "$src" "$dst"
        echo "  Updated: $dst"
        CHANGES=$((CHANGES + 1))
    fi
}

echo "Syncing agent definitions..."
for agent_dir in "$AGENTS_DIR"/*/; do
    agent_name=$(basename "$agent_dir")

    # Sync to repo config/ directory.
    mkdir -p "${CONFIG_DIR}/agents/${agent_name}"
    for f in "$agent_dir"*.{json,md,yaml} 2>/dev/null; do
        [[ -f "$f" ]] || continue
        sync_file "$f" "${CONFIG_DIR}/agents/${agent_name}/$(basename "$f")"
    done

    # Sync to mounted runtime directory.
    runtime_agent="${RUNTIME_DIR}/agents/${agent_name}"
    mkdir -p "$runtime_agent"
    for f in "$agent_dir"*.{json,md,yaml} 2>/dev/null; do
        [[ -f "$f" ]] || continue
        sync_file "$f" "${runtime_agent}/$(basename "$f")"
    done
done

# --- Sync gateway configuration ---
echo "Syncing gateway configuration..."
sync_file "$GATEWAY_CONFIG" "${CONFIG_DIR}/gateway.json"
sync_file "$GATEWAY_CONFIG" "${RUNTIME_DIR}/gateway.json"

# --- Inject secrets into runtime config ---
if [[ -f "${RUNTIME_DIR}/gateway.json" ]]; then
    # Use jq to inject secrets without exposing them in the repo copy.
    jq --arg key "$ANTHROPIC_API_KEY" \
       --arg token "$DISCORD_BOT_TOKEN" \
       '.anthropic_key = $key | .discord_token = $token' \
       "${RUNTIME_DIR}/gateway.json" > "${RUNTIME_DIR}/gateway.json.tmp"
    mv "${RUNTIME_DIR}/gateway.json.tmp" "${RUNTIME_DIR}/gateway.json"
fi

# --- Conditional gateway restart ---
if [[ $CHANGES -gt 0 ]]; then
    echo "$CHANGES file(s) updated. Restarting gateway..."
    if systemctl is-active --quiet agent-gateway 2>/dev/null; then
        systemctl reload agent-gateway
        echo "Gateway reloaded."
    elif [[ -f "${RUNTIME_DIR}/gateway.pid" ]]; then
        kill -HUP "$(cat "${RUNTIME_DIR}/gateway.pid")" 2>/dev/null || true
        echo "Sent HUP to gateway process."
    else
        echo "WARNING: No running gateway found. Start it manually."
    fi
else
    echo "No changes detected. Gateway restart skipped."
fi

Practical Applications

In a multi-agent deployment, these sync scripts are the primary deployment mechanism. When a developer updates an agent’s prompt or adjusts routing rules in gateway.json, they commit the change to the repository and run sync-runtime.sh on the server. The script detects the changed files, copies them to the runtime directory, injects the server’s API credentials, and reloads the gateway — all in under a second for typical changes.

The setup script is run once when provisioning a new server or when a new developer joins. It validates that all required credentials are configured before any agent code runs, preventing the frustrating debugging session where an agent starts, processes its first request, and fails with a cryptic “invalid API key” error buried in logs.

CI pipelines commonly use these scripts for integration testing: the pipeline runs setup.sh with test credentials, deploys agent configurations to a temporary runtime directory, starts the gateway, and runs end-to-end tests against it. The same scripts that run in production run in CI, ensuring that the deployment path is tested on every commit.

Implementation Patterns

Dry-Run Mode

Add a --dry-run flag that reports what would change without making modifications. This lets operators preview a sync before applying it, which is especially valuable when multiple agents are being updated simultaneously and you want to verify the scope of changes.

DRY_RUN="${1:-}"
sync_file() {
    local src="$1" dst="$2"
    if [[ ! -f "$dst" ]] || ! cmp -s "$src" "$dst"; then
        if [[ "$DRY_RUN" == "--dry-run" ]]; then
            echo "  Would update: $dst"
        else
            cp "$src" "$dst"
            echo "  Updated: $dst"
        fi
        CHANGES=$((CHANGES + 1))
    fi
}

Backup Before Overwrite

Before overwriting a runtime configuration file, create a timestamped backup. If a sync introduces a broken configuration, rolling back is a single cp command rather than a git archaeology exercise to find the last known-good version.

Lock File for Concurrent Protection

If multiple operators or CI jobs might run the sync script simultaneously, use a lock file to prevent concurrent execution. Two syncs running at the same time could produce inconsistent state if one writes gateway.json while the other is injecting secrets into it.

LOCK_FILE="${RUNTIME_DIR}/.sync.lock"
exec 200>"$LOCK_FILE"
if ! flock -n 200; then
    echo "ERROR: Another sync is in progress. Exiting."
    exit 1
fi

Checksum Manifest

After a successful sync, write a manifest of file checksums to the runtime directory. The next sync can compare against this manifest instead of reading every runtime file, making the sync faster for large deployments with many agents.

Common Pitfalls

Secrets in Source Control

The .env file must never be committed to git. Add it to .gitignore and provide a .env.example with placeholder values. The sync script intentionally keeps secrets out of the config/ directory (which is committed) and only injects them into the runtime directory. If a developer accidentally commits an .env file, the credentials must be rotated immediately — removing the file from git history is not sufficient.

Stale Runtime State After Agent Removal

When an agent is deleted from the agents/ directory, the sync script doesn’t remove its runtime files or workspace. Orphaned agent directories can confuse the gateway if it scans the runtime directory for available agents. Add a cleanup step that removes runtime directories for agents that no longer exist in the canonical source.

Partial Sync on Script Failure

If the script fails midway (e.g., disk full while copying the third agent), some agents have updated configuration while others have stale files. The set -e flag stops execution but doesn’t roll back already-copied files. For atomic syncs, copy to a staging directory first, then atomically swap it with the runtime directory using mv.

Token Expiration Between Validation and Use

The setup script validates tokens at configuration time, but tokens can expire or be revoked before the agent uses them. Short-lived tokens (like OAuth access tokens) should be refreshed at runtime, not baked into configuration. The validation step catches misconfiguration, but it’s not a substitute for runtime token management.

Further Reading