The Embeddable Go Evaluation Runner is a reusable component for orchestrating LLM evaluation suites. It drives a set of evaluator functions across a corpus of test examples, managing concurrency with rate limiting, collecting per-evaluator statistics, and recording per-example failures—all without aborting the suite on individual failures. The runner is designed to be embedded into larger testing and CI pipelines as a library, not as a standalone binary.
The runner supports two modes of operation: live mode, where it calls a TargetFunc callback to generate fresh outputs for each example, and replay mode, where it reads pre-recorded outputs from disk. Replay mode enables deterministic re-evaluation of historical outputs when evaluator logic changes, without needing to re-invoke the (potentially expensive) LLM.
A critical design goal is resilience: a single failing example or evaluator must not abort the entire suite. The runner captures panics, logs failures with full context, and continues to the next example. At the end, it produces an aggregate report showing pass rates, score distributions, and failure reasons per evaluator, making it straightforward to identify regressions across evaluation runs.
The TargetFunc is a user-provided function with the signature func(ctx context.Context, input Example) (Output, error). It represents the system under evaluation—typically an LLM inference call. The runner invokes this function for each example in the dataset, passing the result to all registered evaluators. In replay mode, the TargetFunc is replaced with a loader that reads outputs from a recorded dataset.
The runner uses a combination of a worker pool (bounded goroutine count) and a token-bucket rate limiter to control the pace of TargetFunc invocations. This prevents overwhelming external LLM APIs with burst traffic while still achieving parallelism. The rate limit is configurable per-runner and is typically set to match the API’s requests-per-minute quota divided by a safety margin.
Each evaluator produces a result (pass/fail, numeric score, or structured judgment) for each example. The runner aggregates these into per-evaluator statistics: pass rate, mean score, score standard deviation, and a histogram of failure reasons. These aggregates are the primary output of an evaluation run and are used to detect regressions when comparing against baseline runs.
When an evaluator fails on a specific example, the runner records the full context: the input, the output, the evaluator’s judgment, and any error. These records are written to a structured log (JSON Lines) that can be inspected post-hoc. Crucially, the runner does not abort on failure—it marks the example as failed and moves to the next one.
In replay mode, the runner skips TargetFunc invocations and instead loads pre-recorded (input, output) pairs from disk. This enables re-evaluating the same outputs with updated evaluator logic, which is essential when tuning evaluation prompts or adding new evaluators. Replay runs are fast (no LLM calls) and deterministic (same inputs, same outputs).
The runner’s execution flow is: load examples, optionally invoke TargetFunc for each, fan out to evaluators, aggregate results, and write the report. Rate limiting and error handling are woven throughout.
package eval
import (
"context"
"fmt"
"sync"
"time"
"golang.org/x/time/rate"
)
type Example struct {
ID string
Input map[string]interface{}
}
type Output struct {
Response string
Metadata map[string]interface{}
}
type EvalResult struct {
Pass bool
Score float64
Reason string
Keywords []string
}
type TargetFunc func(ctx context.Context, ex Example) (Output, error)
type Evaluator interface {
Name() string
Evaluate(ctx context.Context, ex Example, out Output) (EvalResult, error)
}
type RunnerConfig struct {
MaxConcurrency int
RateLimit float64 // requests per second
Timeout time.Duration
}
type Runner struct {
cfg RunnerConfig
target TargetFunc
evaluators []Evaluator
limiter *rate.Limiter
}
func NewRunner(cfg RunnerConfig, target TargetFunc, evaluators ...Evaluator) *Runner {
return &Runner{
cfg: cfg,
target: target,
evaluators: evaluators,
limiter: rate.NewLimiter(rate.Limit(cfg.RateLimit), 1),
}
}type ExampleResult struct {
ExampleID string
Output Output
Evals map[string]EvalResult
Error error
}
func (r *Runner) Run(ctx context.Context, examples []Example) []ExampleResult {
results := make([]ExampleResult, len(examples))
sem := make(chan struct{}, r.cfg.MaxConcurrency)
var wg sync.WaitGroup
for i, ex := range examples {
wg.Add(1)
sem <- struct{}{} // acquire slot
go func(idx int, example Example) {
defer wg.Done()
defer func() { <-sem }() // release slot
result := ExampleResult{
ExampleID: example.ID,
Evals: make(map[string]EvalResult),
}
// Rate limit before calling target
if err := r.limiter.Wait(ctx); err != nil {
result.Error = fmt.Errorf("rate limiter: %w", err)
results[idx] = result
return
}
// Call target function with timeout
tctx, cancel := context.WithTimeout(ctx, r.cfg.Timeout)
defer cancel()
out, err := r.safeCallTarget(tctx, example)
if err != nil {
result.Error = err
results[idx] = result
return
}
result.Output = out
// Run all evaluators
for _, ev := range r.evaluators {
evalResult, evalErr := r.safeCallEvaluator(tctx, ev, example, out)
if evalErr != nil {
result.Evals[ev.Name()] = EvalResult{
Pass: false,
Reason: evalErr.Error(),
}
} else {
result.Evals[ev.Name()] = evalResult
}
}
results[idx] = result
}(i, ex)
}
wg.Wait()
return results
}
func (r *Runner) safeCallTarget(ctx context.Context, ex Example) (out Output, err error) {
defer func() {
if p := recover(); p != nil {
err = fmt.Errorf("panic in TargetFunc: %v", p)
}
}()
return r.target(ctx, ex)
}
func (r *Runner) safeCallEvaluator(
ctx context.Context, ev Evaluator, ex Example, out Output,
) (result EvalResult, err error) {
defer func() {
if p := recover(); p != nil {
err = fmt.Errorf("panic in evaluator %s: %v", ev.Name(), p)
}
}()
return ev.Evaluate(ctx, ex, out)
}import "math"
type EvalStats struct {
Name string
Total int
Passed int
PassRate float64
MeanScore float64
StdDev float64
Failures []FailureRecord
}
type FailureRecord struct {
ExampleID string
Reason string
}
func Aggregate(results []ExampleResult) map[string]*EvalStats {
stats := make(map[string]*EvalStats)
for _, r := range results {
if r.Error != nil {
continue
}
for name, evalResult := range r.Evals {
s, ok := stats[name]
if !ok {
s = &EvalStats{Name: name}
stats[name] = s
}
s.Total++
if evalResult.Pass {
s.Passed++
} else {
s.Failures = append(s.Failures, FailureRecord{
ExampleID: r.ExampleID,
Reason: evalResult.Reason,
})
}
s.MeanScore += evalResult.Score
}
}
// Finalize aggregates
for _, s := range stats {
if s.Total > 0 {
s.PassRate = float64(s.Passed) / float64(s.Total)
s.MeanScore /= float64(s.Total)
}
// Compute standard deviation (second pass)
var variance float64
for _, r := range results {
if r.Error != nil {
continue
}
if ev, ok := r.Evals[s.Name]; ok {
diff := ev.Score - s.MeanScore
variance += diff * diff
}
}
if s.Total > 1 {
s.StdDev = math.Sqrt(variance / float64(s.Total-1))
}
}
return stats
}In a typical Go-based AI platform, the evaluation runner is embedded into the CI pipeline to run regression tests against LLM-generated outputs. Each pull request triggers an evaluation suite that compares the PR branch’s output quality against a baseline. If any evaluator’s pass rate drops below a configured threshold, the CI check fails and the PR cannot be merged.
Replay mode is used when the team tunes evaluation prompts. Rather than re-running expensive LLM inference, they replay the last production run’s recorded outputs through updated evaluators. This lets them iterate on evaluation criteria quickly—a replay run of 1,000 examples takes seconds instead of the minutes or hours required for live inference.
The per-example failure records feed into a dashboard that highlights the most common failure modes. Product teams use this to prioritize prompt engineering work: if 30% of failures share the same reason string (“response exceeded token limit”), that becomes the next optimization target.
ctx.Done() and returns promptly. Goroutines stuck in blocking API calls will leak until the process exits.