Running evaluations against LLM APIs at scale presents a fundamental tension: you want maximum throughput to finish evaluation runs quickly, but API providers enforce rate limits that throttle or reject requests when you exceed their quotas. A naive approach — fire all requests concurrently — results in a burst of 429 (Too Many Requests) responses, wasted tokens, and evaluation runs that take longer than a properly paced run would have.
The evaluation framework’s Runner solves this with a layered concurrency control system: a semaphore caps overall concurrency (how many evaluations can be in-flight simultaneously), while per-evaluator token-bucket rate limiters enforce API-specific quotas (e.g., 60 requests per minute for one LLM provider, 200 for another). Goroutines acquire both the semaphore lease and a rate limiter token before executing, ensuring that the system never exceeds either the global concurrency ceiling or any individual provider’s rate limit.
This pattern is broadly applicable beyond LLM evaluation — any system that orchestrates concurrent work against rate-limited external services (payment gateways, search APIs, notification providers) faces the same design challenge. The key insight is that concurrency limiting and rate limiting are orthogonal concerns that must be composed, not conflated.
A semaphore is a counter that controls how many goroutines can execute a critical section concurrently. Go’s golang.org/x/sync/semaphore package provides a weighted semaphore: Acquire(ctx, 1) blocks until a slot is available, and Release(1) frees it. By setting MaxConcurrency to, say, 10, the Runner ensures at most 10 evaluation calls are in-flight at any moment — regardless of how many dataset items are queued. This prevents resource exhaustion (memory, file descriptors, goroutines) even when evaluating thousands of items.
A token bucket refills at a steady rate (e.g., 1 token per second for a 60 RPM limit) and holds up to a burst capacity. Each request consumes one token; if the bucket is empty, the request blocks until a token becomes available. Go’s golang.org/x/time/rate package implements this as rate.NewLimiter(rate.Limit(rps), burst). Unlike simple sleep-between-requests approaches, token buckets allow controlled bursts while maintaining long-term rate compliance — critical for APIs that allow short bursts but enforce sustained rate limits.
Different LLM providers have different rate limits. OpenAI might allow 3,500 RPM on one tier, while Anthropic’s Claude API has a different quota. The Runner associates a separate rate.Limiter with each evaluator, configured from the evaluation spec. This means a single evaluation run can fan out to multiple providers simultaneously, each respecting its own rate limit, without one provider’s quota affecting another.
When running evaluations at scale, knowing that “47 of 200 evaluations failed” isn’t actionable — you need to know which evaluator failed and why. The Runner tracks success counts, error counts, and error categories per evaluator. After a run completes, the stats reveal whether failures are concentrated in one provider (suggesting an API issue) or spread evenly (suggesting a dataset problem). This per-evaluator accounting is maintained with atomic counters to avoid lock contention in the hot path.
When a rate limiter blocks, the goroutine waits — it doesn’t fail. This is deliberate: transient rate limit pressure should slow the run, not abort it. However, if the context is cancelled (e.g., the user presses Ctrl+C or a timeout fires), blocked goroutines unblock immediately with a context error. The Runner distinguishes between “slow because rate-limited” (normal) and “stuck because cancelled” (abort), ensuring clean shutdown without orphaned goroutines.
The Runner processes a dataset by spawning one goroutine per item, gated by the semaphore. Each goroutine acquires its evaluator’s rate limiter token before making the API call. Results flow back through a channel, and statistics are updated atomically.
package runner
import (
"context"
"sync"
"sync/atomic"
"golang.org/x/sync/semaphore"
"golang.org/x/time/rate"
)
type Runner struct {
maxConcurrency int64
evaluators map[string]*EvaluatorConfig
limiters map[string]*rate.Limiter
stats map[string]*EvalStats
}
type EvaluatorConfig struct {
Name string
RateLimit float64 // requests per second
BurstSize int
Handler EvalHandler
}
type EvalStats struct {
Success atomic.Int64
Errors atomic.Int64
}
type EvalHandler func(ctx context.Context, input string) (string, error)
func NewRunner(maxConcurrency int, configs []*EvaluatorConfig) *Runner {
r := &Runner{
maxConcurrency: int64(maxConcurrency),
evaluators: make(map[string]*EvaluatorConfig),
limiters: make(map[string]*rate.Limiter),
stats: make(map[string]*EvalStats),
}
for _, cfg := range configs {
r.evaluators[cfg.Name] = cfg
r.limiters[cfg.Name] = rate.NewLimiter(
rate.Limit(cfg.RateLimit),
cfg.BurstSize,
)
r.stats[cfg.Name] = &EvalStats{}
}
return r
}type DatasetItem struct {
ID string
Input string
Evaluator string // which evaluator to use
}
type Result struct {
ItemID string
Output string
Error error
Evaluator string
}
func (r *Runner) Run(ctx context.Context, items []DatasetItem) []Result {
sem := semaphore.NewWeighted(r.maxConcurrency)
results := make([]Result, len(items))
var wg sync.WaitGroup
for i, item := range items {
wg.Add(1)
go func(idx int, item DatasetItem) {
defer wg.Done()
// Acquire semaphore slot — blocks if at max concurrency.
if err := sem.Acquire(ctx, 1); err != nil {
results[idx] = Result{
ItemID: item.ID, Error: err, Evaluator: item.Evaluator,
}
return
}
defer sem.Release(1)
// Acquire rate limiter token — blocks until rate allows.
limiter := r.limiters[item.Evaluator]
if err := limiter.Wait(ctx); err != nil {
results[idx] = Result{
ItemID: item.ID, Error: err, Evaluator: item.Evaluator,
}
return
}
// Execute the evaluation.
handler := r.evaluators[item.Evaluator].Handler
output, err := handler(ctx, item.Input)
// Update per-evaluator stats.
stats := r.stats[item.Evaluator]
if err != nil {
stats.Errors.Add(1)
} else {
stats.Success.Add(1)
}
results[idx] = Result{
ItemID: item.ID,
Output: output,
Error: err,
Evaluator: item.Evaluator,
}
}(i, item)
}
wg.Wait()
return results
}// ParseRateLimit converts human-readable rate specs to rate.Limit values.
// Supports formats like "60/min", "1000/hour", "10/sec".
func ParseRateLimit(spec string) (rate.Limit, int, error) {
parts := strings.Split(spec, "/")
if len(parts) != 2 {
return 0, 0, fmt.Errorf("invalid rate spec %q: expected N/unit", spec)
}
count, err := strconv.ParseFloat(parts[0], 64)
if err != nil {
return 0, 0, fmt.Errorf("invalid count in rate spec %q: %w", spec, err)
}
var rps float64
switch strings.ToLower(parts[1]) {
case "sec", "second":
rps = count
case "min", "minute":
rps = count / 60.0
case "hour":
rps = count / 3600.0
default:
return 0, 0, fmt.Errorf("unknown time unit %q in rate spec", parts[1])
}
// Burst size defaults to 10% of the per-second rate, minimum 1.
burst := int(math.Max(1, math.Ceil(rps*0.1)))
return rate.Limit(rps), burst, nil
}The evaluation framework uses this Runner to orchestrate LLM-as-judge evaluation pipelines. A typical evaluation run sends hundreds of prompts to one or more LLM APIs, collects their responses, and scores them against a rubric. Without rate limiting, a 500-item evaluation against a provider with a 60 RPM limit would hit the rate limit within the first second, resulting in retries, backoff delays, and a run that takes 3x longer than necessary.
With the Runner’s rate limiting, the same 500-item evaluation proceeds at a steady 1 request per second to the rate-limited provider. If another evaluator in the same run has a 1,000 RPM limit, it runs at ~16.7 requests per second — both evaluators operate independently within their quotas. The semaphore prevents more than MaxConcurrency goroutines from being active simultaneously, bounding memory usage regardless of dataset size.
The per-evaluator statistics have proven especially valuable for debugging evaluation pipelines. When a run shows 95% success on one evaluator but 60% on another, the stats immediately narrow the investigation to the failing provider — is it returning errors, timing out, or producing malformed responses? This level of observability turns a “the evaluation failed” report into an actionable diagnosis.
The token bucket controls the request rate proactively, but APIs can still return 429 responses if the rate limit is misconfigured or the provider lowers limits dynamically. A robust runner wraps the handler with retry logic that detects 429 responses, extracts the Retry-After header, and temporarily reduces the rate limiter’s rate to avoid further rejections.
func withRetry(handler EvalHandler, limiter *rate.Limiter) EvalHandler {
return func(ctx context.Context, input string) (string, error) {
maxRetries := 3
for attempt := 0; attempt <= maxRetries; attempt++ {
output, err := handler(ctx, input)
if err == nil {
return output, nil
}
var apiErr *APIError
if !errors.As(err, &apiErr) || apiErr.StatusCode != 429 {
return "", err // Non-retryable error
}
// Back off: reduce the limiter's rate temporarily.
currentLimit := limiter.Limit()
limiter.SetLimit(currentLimit * 0.5)
retryAfter := apiErr.RetryAfter
if retryAfter == 0 {
retryAfter = time.Duration(attempt+1) * 2 * time.Second
}
select {
case <-time.After(retryAfter):
// Restore rate after backoff.
limiter.SetLimit(currentLimit)
case <-ctx.Done():
return "", ctx.Err()
}
}
return "", fmt.Errorf("max retries exceeded")
}
}For long-running evaluation runs, a progress callback reports completion percentage, current throughput, and estimated time remaining. The Runner invokes this callback after each item completes, passing the cumulative stats. The CLI renders this as a progress bar; the web UI renders it as a live dashboard.
Individual evaluations should have their own timeout separate from the overall run timeout. A single slow LLM response shouldn’t block the entire semaphore slot indefinitely. Wrap each handler call in a context with a per-evaluation timeout derived from the evaluator config.
Setting MaxConcurrency=60 does not mean 60 requests per minute. It means 60 requests can be in-flight simultaneously. If each request takes 2 seconds, you’ll make 30 requests per second — 1,800 per minute. Concurrency limits control parallelism; rate limits control throughput over time. You need both: the semaphore prevents resource exhaustion, and the rate limiter prevents API abuse.
If the semaphore is sized too small relative to the number of evaluators, high-throughput evaluators can starve slower ones by occupying all semaphore slots. For example, with MaxConcurrency=5 and two evaluators where one has a 100 RPM limit and the other has 10 RPM, the fast evaluator may grab all 5 slots before the slow one gets any. Consider per-evaluator semaphore partitioning or weighted fair queuing if this becomes an issue.
Both semaphore.Acquire and rate.Limiter.Wait accept a context. If you pass context.Background() instead of the run’s cancellable context, Ctrl+C won’t stop blocked goroutines — they’ll wait indefinitely for a semaphore slot or rate token. Always propagate the cancellable context through the entire call chain.
A token bucket with a burst size of 100 and a rate of 1/second will allow 100 requests to fire instantly before settling into the 1/second pace. If the API’s rate limit window is shorter than your burst (e.g., a 10-second sliding window), the burst can still trigger rate limiting. Set burst sizes conservatively — typically 1-5 for strict rate limits, or 10-20% of the per-second rate for more lenient APIs.