When an AI agent uses tools to accomplish tasks, evaluating its behavior requires more than checking whether the final answer is correct. Two agents might arrive at the same answer but through very different tool call sequences — one efficient and methodical, the other wasteful and circuitous. Agent evaluation frameworks need to assess both what tools the agent selected and the order in which it called them, comparing actual behavior against expected behavior defined by human evaluators.
As an example, an agent evaluation framework might implement evaluators that normalize tool names across different representations (strings, maps, structured objects), compare expected versus actual tool sets using overlap scoring, and run longest common subsequence (LCS) checks to grade tool call ordering. These evaluators answer questions like: “Did the agent use the right tools?” (selection accuracy), “Did it use them in a reasonable order?” (sequence accuracy), and “Did it avoid unnecessary calls?” (efficiency).
This topic covers the algorithms and patterns behind agent tool auditing: tool name normalization for consistent comparison, set-based overlap scoring for selection accuracy, longest common subsequence for sequence grading, and the evaluation framework architecture that composes these checks into comprehensive agent assessments. These techniques are essential for building reliable agent evaluation pipelines that catch regressions in agent behavior as models, prompts, or tool definitions change.
Agent frameworks represent tool calls in different formats: some use plain strings ("search_documents"), others use maps ({"name": "search_documents", "id": "call_123"}), and others use typed structs with additional metadata. Before any comparison, tool names must be extracted and normalized to a canonical form. Normalization strips whitespace, lowercases names, and extracts the tool name from wrapper objects. Without normalization, "Search_Documents" and "search_documents" would count as different tools, producing false negatives in evaluation.
Selection accuracy measures whether the agent called the right set of tools, regardless of order. The overlap score is computed as the size of the intersection divided by the size of the union (Jaccard similarity) or divided by the expected set size (recall). A score of 1.0 means the agent called exactly the expected tools with no extras and no omissions. Scores below 1.0 can be decomposed into missing tools (recall failures) and extra tools (precision failures), giving evaluators specific signal about what went wrong.
The LCS algorithm finds the longest sequence of elements that appear in the same relative order in both the expected and actual tool call sequences, without requiring them to be contiguous. An LCS length equal to the expected sequence length means the agent called tools in exactly the right order (possibly with extra calls interspersed). The LCS ratio (LCS length / expected length) gives a 0-to-1 score for ordering accuracy. This is more forgiving than exact sequence matching: if the agent makes one extra call between two correct ones, the LCS score remains high.
A comprehensive agent evaluation combines multiple dimensions: tool selection (did it pick the right tools?), tool ordering (did it call them in a sensible sequence?), argument accuracy (did it pass correct parameters?), and outcome correctness (did the final answer match expectations?). Each dimension produces a score between 0 and 1, and the overall evaluation can weight these dimensions differently based on what matters most. For some tasks, getting the right answer with wrong tool order is acceptable; for others, the process matters as much as the result.
An agent evaluation framework like this organizes evaluators as composable functions that take an agent trace (the recorded sequence of tool calls and responses) and an expectation (the human-defined correct behavior) and return a scored result. Evaluators are registered by name and can be combined: a “tool_selection” evaluator checks the set, a “tool_ordering” evaluator checks the sequence, and a “composite” evaluator aggregates both scores. This modular architecture lets teams add new evaluation dimensions without modifying existing evaluators.
The evaluation pipeline processes an agent trace through three stages: normalization, comparison, and scoring.
The raw agent trace contains tool calls in the format emitted by the specific agent framework — this might be a list of function call objects, a JSON transcript, or structured log entries. The normalizer extracts the tool name from each call, applies case folding and whitespace trimming, and produces a clean list of tool name strings. It also handles aliases: if the evaluation config maps "search" to "search_documents", both forms are treated as the same tool. This makes evaluations robust against minor naming changes in tool definitions.
The normalized actual tool list is compared against the expected tool list from the evaluation case. Two comparisons run in parallel: (1) a set comparison that ignores order and computes intersection, union, missing, and extra tools, and (2) a sequence comparison that runs the LCS algorithm to find the longest common subsequence. Both comparisons produce intermediate data structures that the scoring stage consumes.
The scoring stage converts comparison results into numeric scores. The selection score uses the Jaccard index (intersection size / union size) or recall (intersection size / expected size), depending on configuration. The ordering score divides the LCS length by the expected sequence length. A weighted average combines these into an overall tool use score. The evaluator also produces a human-readable explanation: “Agent called 4/5 expected tools (missed: validate_input) and called them in 80% correct order (swapped analyze and transform).”
Normalize tool names from various representations into a canonical string form for consistent comparison.
package evaluators
import (
"strings"
)
// NormalizeToolName extracts and normalizes a tool name from various formats.
// Handles strings, maps with "name" keys, and structs with Name fields.
func NormalizeToolName(raw any) string {
switch v := raw.(type) {
case string:
return strings.ToLower(strings.TrimSpace(v))
case map[string]any:
if name, ok := v["name"]; ok {
return NormalizeToolName(name)
}
if name, ok := v["function"]; ok {
return NormalizeToolName(name)
}
return ""
default:
return ""
}
}
// NormalizeToolList extracts tool names from a list of tool call records.
func NormalizeToolList(calls []any) []string {
var names []string
for _, call := range calls {
name := NormalizeToolName(call)
if name != "" {
names = append(names, name)
}
}
return names
}
// ApplyAliases maps aliased tool names to their canonical forms.
func ApplyAliases(names []string, aliases map[string]string) []string {
result := make([]string, len(names))
for i, name := range names {
if canonical, ok := aliases[name]; ok {
result[i] = canonical
} else {
result[i] = name
}
}
return result
}Compute selection accuracy via set overlap and ordering accuracy via longest common subsequence.
package evaluators
// ToolOverlapScore computes the Jaccard similarity between expected and actual tool sets.
// Returns a score between 0.0 and 1.0, plus the missing and extra tool names.
func ToolOverlapScore(expected, actual []string) (score float64, missing, extra []string) {
expectedSet := make(map[string]bool)
actualSet := make(map[string]bool)
for _, t := range expected {
expectedSet[t] = true
}
for _, t := range actual {
actualSet[t] = true
}
// Intersection
var intersectionSize int
for t := range expectedSet {
if actualSet[t] {
intersectionSize++
} else {
missing = append(missing, t)
}
}
// Extra tools not in expected
for t := range actualSet {
if !expectedSet[t] {
extra = append(extra, t)
}
}
// Jaccard index: |A ∩ B| / |A ∪ B|
unionSize := len(expectedSet) + len(actualSet) - intersectionSize
if unionSize == 0 {
return 1.0, nil, nil // both empty
}
score = float64(intersectionSize) / float64(unionSize)
return score, missing, extra
}
// LCSLength computes the length of the longest common subsequence.
func LCSLength(expected, actual []string) int {
m, n := len(expected), len(actual)
if m == 0 || n == 0 {
return 0
}
// DP table: dp[i][j] = LCS length of expected[:i] and actual[:j]
dp := make([][]int, m+1)
for i := range dp {
dp[i] = make([]int, n+1)
}
for i := 1; i <= m; i++ {
for j := 1; j <= n; j++ {
if expected[i-1] == actual[j-1] {
dp[i][j] = dp[i-1][j-1] + 1
} else {
dp[i][j] = max(dp[i-1][j], dp[i][j-1])
}
}
}
return dp[m][n]
}
// ToolOrderScore computes the ordering accuracy as LCS length / expected length.
func ToolOrderScore(expected, actual []string) float64 {
if len(expected) == 0 {
return 1.0
}
lcs := LCSLength(expected, actual)
return float64(lcs) / float64(len(expected))
}
func max(a, b int) int {
if a > b {
return a
}
return b
}Combines selection and ordering scores into a weighted overall evaluation result.
package evaluators
import "fmt"
type EvalResult struct {
SelectionScore float64 `json:"selection_score"`
OrderingScore float64 `json:"ordering_score"`
OverallScore float64 `json:"overall_score"`
MissingTools []string `json:"missing_tools,omitempty"`
ExtraTools []string `json:"extra_tools,omitempty"`
Explanation string `json:"explanation"`
}
type EvalConfig struct {
SelectionWeight float64 `json:"selection_weight"` // default 0.6
OrderingWeight float64 `json:"ordering_weight"` // default 0.4
Aliases map[string]string `json:"aliases,omitempty"`
}
func EvaluateToolUse(
expectedCalls []any,
actualCalls []any,
config EvalConfig,
) EvalResult {
// Normalize both lists
expected := NormalizeToolList(expectedCalls)
actual := NormalizeToolList(actualCalls)
// Apply aliases if configured
if len(config.Aliases) > 0 {
expected = ApplyAliases(expected, config.Aliases)
actual = ApplyAliases(actual, config.Aliases)
}
// Compute scores
selScore, missing, extra := ToolOverlapScore(expected, actual)
ordScore := ToolOrderScore(expected, actual)
overall := config.SelectionWeight*selScore + config.OrderingWeight*ordScore
explanation := fmt.Sprintf(
"Selection: %.0f%% (called %d/%d expected tools",
selScore*100,
len(expected)-len(missing),
len(expected),
)
if len(missing) > 0 {
explanation += fmt.Sprintf(", missed: %v", missing)
}
if len(extra) > 0 {
explanation += fmt.Sprintf(", extra: %v", extra)
}
explanation += fmt.Sprintf("). Ordering: %.0f%%.", ordScore*100)
return EvalResult{
SelectionScore: selScore,
OrderingScore: ordScore,
OverallScore: overall,
MissingTools: missing,
ExtraTools: extra,
Explanation: explanation,
}
}When a prompt is modified to improve one agent behavior, it may degrade another. The tool use evaluator runs across a suite of test cases after each prompt change, catching regressions where the agent starts calling wrong tools or calling them in suboptimal order. A CI pipeline can gate prompt changes on maintaining a minimum tool use score across the evaluation suite, preventing prompt regressions from reaching production.
When evaluating whether to upgrade from one model to another (e.g., GPT-4 to GPT-4 Turbo, or Llama 2 to Llama 3), the tool use evaluator provides a structured comparison. Run the same agent tasks on both models, score each model’s tool use, and compare selection accuracy, ordering accuracy, and overall scores. This gives objective data beyond “the answers look good” — it reveals whether the new model is more or less efficient in its tool use patterns.
During agent development, evaluators provide immediate feedback on whether changes to tool definitions, system prompts, or chain logic improve or degrade tool use quality. A developer modifying a tool schema can run the evaluator before and after the change, seeing exactly how the agent’s tool selection and ordering changed. The human-readable explanations (“missed: validate_input, extra: retry_search”) point directly at what to fix.