YAML-Driven Evaluator Pipelines

Overview

Evaluation pipelines for LLM applications involve many moving parts: which models to evaluate, what datasets to run against, how to score outputs, where to store results, and what execution constraints to apply (parallelism, timeouts, rate limits). Hardcoding these parameters into application code makes experimentation painful — every change requires a code modification, recompilation, and redeployment. Worse, it makes evaluation runs non-reproducible: the parameters that produced last week’s results exist only in someone’s local branch.

A YAML-driven evaluation tool addresses this by making YAML the single source of truth for evaluation configuration. A versioned YAML file declares the complete evaluation specification: evaluators (which LLM judges to use), datasets (what inputs to evaluate), output targets (where results go), and execution knobs (parallelism, timeouts, rate limits). The CLI reads this file, constructs the evaluation pipeline, and executes it. Runtime overrides allow ad-hoc experimentation — run a specific evaluator against a subset of the dataset — without modifying the checked-in config.

This declarative approach transforms evaluation from an imperative script (“run this function with these arguments”) into a reproducible specification (“this is what should happen”). The YAML file can be code-reviewed, diffed, and versioned alongside the code it evaluates, creating a complete audit trail of what was evaluated, how, and when.

Core Concepts

How It Works

The evaluation CLI parses the YAML config, resolves runtime overrides, constructs the evaluation pipeline, and delegates execution to the Runner. The config struct mirrors the YAML schema, with validation at parse time to catch misconfigurations before any evaluation work begins.

YAML Configuration Schema

# eval/config.yaml
version: "1"
name: "weekly-quality-eval"
description: "Evaluate response quality across all active models"

execution:
  parallelism: 10
  timeout: 30m
  retry:
    max_attempts: 3
    backoff: exponential

datasets:
  - name: "golden-set"
    path: "eval/datasets/golden-v3.jsonl"
    format: jsonl
  - name: "edge-cases"
    path: "eval/datasets/edge-cases.jsonl"
    format: jsonl

evaluators:
  - name: "gpt4-factuality"
    model: "gpt-4-turbo"
    prompt_template: "eval/prompts/factuality.txt"
    rate_limit: "60/min"
    timeout: 30s
    parameters:
      temperature: 0.0
      max_tokens: 512

  - name: "claude-style"
    model: "claude-3-sonnet"
    prompt_template: "eval/prompts/style.txt"
    rate_limit: "100/min"
    timeout: 45s
    parameters:
      temperature: 0.0
      max_tokens: 256

  - name: "rubric-scorer"
    model: "gpt-4-turbo"
    prompt_template: "eval/prompts/rubric.txt"
    rate_limit: "60/min"
    depends_on: ["gpt4-factuality", "claude-style"]

outputs:
  - type: jsonl
    path: "eval/results/{{.RunID}}.jsonl"
  - type: bigquery
    table: "evals.weekly_quality"
    project: "example-analytics"
  - type: slack
    channel: "#eval-results"
    on: complete

Config Parsing and Validation

package config

import (
    "fmt"
    "os"
    "time"

    "gopkg.in/yaml.v3"
)

type EvalConfig struct {
    Version     string            `yaml:"version"`
    Name        string            `yaml:"name"`
    Description string            `yaml:"description"`
    Execution   ExecutionConfig   `yaml:"execution"`
    Datasets    []DatasetConfig   `yaml:"datasets"`
    Evaluators  []EvaluatorConfig `yaml:"evaluators"`
    Outputs     []OutputConfig    `yaml:"outputs"`
}

type ExecutionConfig struct {
    Parallelism int           `yaml:"parallelism"`
    Timeout     time.Duration `yaml:"timeout"`
    Retry       RetryConfig   `yaml:"retry"`
}

type DatasetConfig struct {
    Name   string `yaml:"name"`
    Path   string `yaml:"path"`
    Format string `yaml:"format"`
}

type EvaluatorConfig struct {
    Name           string            `yaml:"name"`
    Model          string            `yaml:"model"`
    PromptTemplate string            `yaml:"prompt_template"`
    RateLimit      string            `yaml:"rate_limit"`
    Timeout        time.Duration     `yaml:"timeout"`
    Parameters     map[string]any    `yaml:"parameters"`
    DependsOn      []string          `yaml:"depends_on"`
}

type OutputConfig struct {
    Type    string         `yaml:"type"`
    Path    string         `yaml:"path,omitempty"`
    Table   string         `yaml:"table,omitempty"`
    Channel string         `yaml:"channel,omitempty"`
    On      string         `yaml:"on,omitempty"`
}

func Load(path string) (*EvalConfig, error) {
    data, err := os.ReadFile(path)
    if err != nil {
        return nil, fmt.Errorf("reading config %s: %w", path, err)
    }

    var cfg EvalConfig
    if err := yaml.Unmarshal(data, &cfg); err != nil {
        return nil, fmt.Errorf("parsing config: %w", err)
    }

    if err := cfg.Validate(); err != nil {
        return nil, fmt.Errorf("validating config: %w", err)
    }

    return &cfg, nil
}

func (c *EvalConfig) Validate() error {
    if len(c.Datasets) == 0 {
        return fmt.Errorf("at least one dataset is required")
    }
    if len(c.Evaluators) == 0 {
        return fmt.Errorf("at least one evaluator is required")
    }
    for _, e := range c.Evaluators {
        if e.Model == "" {
            return fmt.Errorf("evaluator %q: model is required", e.Name)
        }
        if e.PromptTemplate == "" {
            return fmt.Errorf("evaluator %q: prompt_template is required", e.Name)
        }
    }
    return nil
}

CLI Override Application

package main

import (
    "flag"
    "strings"
)

type Overrides struct {
    Evaluators  []string
    Dataset     string
    Parallelism int
    DryRun      bool
}

func parseOverrides() *Overrides {
    o := &Overrides{}
    var evals string
    flag.StringVar(&evals, "evaluator", "", "run only these evaluators (comma-separated)")
    flag.StringVar(&o.Dataset, "dataset", "", "override dataset path")
    flag.IntVar(&o.Parallelism, "parallelism", 0, "override parallelism")
    flag.BoolVar(&o.DryRun, "dry-run", false, "validate config and print plan without executing")
    flag.Parse()

    if evals != "" {
        o.Evaluators = strings.Split(evals, ",")
    }
    return o
}

func applyOverrides(cfg *config.EvalConfig, o *Overrides) {
    if o.Parallelism > 0 {
        cfg.Execution.Parallelism = o.Parallelism
    }
    if o.Dataset != "" {
        cfg.Datasets = []config.DatasetConfig{
            {Name: "override", Path: o.Dataset, Format: "jsonl"},
        }
    }
    if len(o.Evaluators) > 0 {
        filtered := make([]config.EvaluatorConfig, 0)
        allowed := make(map[string]bool)
        for _, name := range o.Evaluators {
            allowed[name] = true
        }
        for _, e := range cfg.Evaluators {
            if allowed[e.Name] {
                filtered = append(filtered, e)
            }
        }
        cfg.Evaluators = filtered
    }
}

Practical Applications

In a YAML-driven evaluation system, configs are checked into the repository alongside the code they evaluate. Each product team maintains its own evaluation config: the search team’s config evaluates retrieval quality with domain-specific judges, while the summarization team’s config measures factual consistency and conciseness. Both configs share the same runner infrastructure but operate independently.

CI integration uses the YAML configs directly. The GitHub Actions workflow runs the evaluation CLI (e.g., eval-cli run eval/ci-config.yaml) on every pull request, comparing results against a baseline stored in the repository. If quality metrics regress beyond a threshold, the PR check fails. The CI config is intentionally smaller than the full weekly evaluation — it uses a subset of the dataset and fewer evaluators to keep PR feedback fast (under 5 minutes).

The --dry-run flag has become a critical part of the evaluation development workflow. Before launching a multi-hour evaluation run against expensive APIs, developers run the CLI with --dry-run to validate the config, verify dataset paths exist, check that prompt templates parse correctly, and print the execution plan. This catches most misconfigurations without spending API credits.

Implementation Patterns

Template Variables in Configs

Output paths and other string values support Go template syntax for dynamic values. {{.RunID}} expands to a unique run identifier, {{.Timestamp}} to the run start time, and {{.GitSHA}} to the current commit hash. This prevents output file collisions and makes results traceable to specific code versions.

Config Inheritance

A base config can define shared defaults (execution parameters, common evaluators), and environment-specific configs can extend it. The CLI supports --base=eval/base.yaml which loads defaults before applying the primary config. This reduces duplication across dev, staging, and production evaluation configs while allowing environment-specific overrides.

Schema Validation

Beyond structural validation, the config loader performs semantic validation: checking that referenced files exist, that evaluator names in depends_on reference declared evaluators, that rate limit strings parse correctly, and that output target credentials are available. Fail-fast validation at load time prevents wasted API calls on misconfigured runs.

Common Pitfalls

YAML Indentation Errors

YAML is whitespace-sensitive, and indentation errors produce confusing parse failures. A common mistake is mixing tabs and spaces, or indenting a nested key at the wrong level. Use a YAML linter in CI (e.g., yamllint) and configure your editor to highlight YAML syntax errors. For complex configs, consider generating YAML from a typed config builder rather than editing it by hand.

Stale Dataset Paths

Renaming or moving a dataset file without updating all evaluation configs that reference it causes runtime failures — often after the config has passed structural validation but during dataset loading. Use the --dry-run flag to catch missing files before launching a run. Some teams add a CI step that validates all .yaml files in the eval/ directory to catch broken references early.

Overriding Too Much at Runtime

The override mechanism is designed for quick iteration, not for production runs. If you find yourself passing 5+ flags to the evaluation CLI, create a new YAML config instead. Overrides aren’t captured in version control, which defeats the reproducibility goal. The --dry-run output includes a “resolved config” section that shows the final merged config — save this to a file if you want to capture an ad-hoc run’s parameters.

Missing depends_on for Chained Evaluators

If evaluator B uses evaluator A’s output as input but doesn’t declare depends_on: [A], the runner may execute them in parallel — and B will fail because A’s results aren’t available yet. The runner validates the dependency graph at startup and topologically sorts evaluators, but only if dependencies are declared. Undeclared dependencies cause non-deterministic failures that are hard to reproduce.

Further Reading