Evaluating AI Agents in Production: Traces, Datasets, and Regression Tests



Evaluating AI Agents in Production: Traces, Datasets, and Regression Tests

Quick Answer / TL;DR

To effectively evaluate AI agents in production, implement a three-pronged strategy: capture detailed **execution traces** to understand agent behavior, transform these traces into structured **evaluation datasets** for analysis, and establish **automated regression tests** using golden datasets and LLM-as-a-judge techniques to continuously monitor performance and detect degradations with every new deployment.

The rise of AI agents, powered by large language models (LLMs), has opened new frontiers in automation and intelligent systems. From customer service chatbots to autonomous code assistants, these agents are increasingly deployed in production environments, making real-world decisions. However, evaluating AI agents in production presents unique challenges. Unlike traditional software, their non-deterministic nature and reliance on dynamic external tools make traditional unit tests insufficient. How do we ensure they consistently perform as expected, maintain quality, and don’t “drift” over time? This guide will walk you through practical, hands-on strategies to build a robust evaluation pipeline using traces, datasets, and automated regression tests.

What You Will Learn

Table of Contents

The Challenge of Evaluating AI Agents in Production

Traditional software testing often relies on deterministic inputs yielding predictable outputs. AI agents, however, introduce several complexities:

  1. Non-Determinism: LLMs can generate different responses to the same prompt, even with identical settings.
  2. Tool Use: Agents interact with external tools (databases, APIs, web search), whose behavior can change or be inconsistent.
  3. Complex Reasoning Paths: An agent’s “thought process” involves multiple steps, tool calls, and LLM inferences, making failure points hard to pinpoint.
  4. Subjective Quality: Evaluating aspects like “helpfulness,” “relevance,” or “coherence” often requires human judgment.
  5. Production Drift: Agent performance can degrade over time due to changes in user queries, underlying LLM updates, or tool API modifications.

To tackle these, we need a system that offers visibility into agent execution, transforms raw interactions into measurable data, and automates the validation of critical functionalities.

Capturing AI Agent Traces for Enhanced Observability

Traces are detailed logs of an AI agent’s internal thought process and interactions. They provide a forensic trail of every decision, tool call, and LLM invocation, which is indispensable for debugging, understanding behavior, and ultimately, evaluating AI agents in production.

Step 1: Instrument Your Agent with Tracing

Most popular agent frameworks offer built-in tracing capabilities. For instance, LangChain provides Callbacks that allow you to hook into various stages of an agent’s execution.

Real-world Example: E-commerce Customer Service Agent Imagine an agent designed to answer product queries, check order status, and process returns. A trace would show:

Here’s how you might instrument a simple LangChain agent:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121

# main.py
import os
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_openai import ChatOpenAI
from langchain import hub
from langchain_core.tools import tool
from langchain.callbacks.base import BaseCallbackHandler
from typing import Any, Dict, List, Optional
import json
import uuid
import datetime

# Mock Tools
@tool
def get_product_details(product_id: str) -> str:
    """Returns details for a specific product ID."""
    if product_id == "PROD123":
        return "Product 123: High-quality widget, in stock, $29.99"
    return "Product not found."

@tool
def check_order_status(order_id: str) -> str:
    """Checks the status of a customer order."""
    if order_id == "ORDER456":
        return "Order 456: Shipped, ETA Aug 30."
    return "Order not found or invalid."

tools = [get_product_details, check_order_status]

# Custom Callback Handler for Tracing
class MyTraceCallbackHandler(BaseCallbackHandler):
    def __init__(self):
        self.trace_id = str(uuid.uuid4())
        self.events = []

    def on_llm_start(
        self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any
    ) -> None:
        self.events.append({
            "timestamp": datetime.datetime.now().isoformat(),
            "type": "llm_start",
            "prompts": prompts,
            "llm_model": serialized.get("kwargs", {}).get("model_name"),
            "trace_id": self.trace_id,
        })

    def on_llm_end(self, response: Any, **kwargs: Any) -> None:
        self.events.append({
            "timestamp": datetime.datetime.now().isoformat(),
            "type": "llm_end",
            "response": response.generations[0][0].text,
            "trace_id": self.trace_id,
        })

    def on_tool_start(
        self, serialized: Dict[str, Any], input_str: str, **kwargs: Any
    ) -> None:
        self.events.append({
            "timestamp": datetime.datetime.now().isoformat(),
            "type": "tool_start",
            "tool_name": serialized["name"],
            "tool_input": input_str,
            "trace_id": self.trace_id,
        })

    def on_tool_end(self, output: str, **kwargs: Any) -> None:
        self.events.append({
            "timestamp": datetime.datetime.now().isoformat(),
            "type": "tool_end",
            "tool_output": output,
            "trace_id": self.trace_id,
        })
    
    def on_agent_action(self, action: Any, **kwargs: Any) -> Any:
        self.events.append({
            "timestamp": datetime.datetime.now().isoformat(),
            "type": "agent_action",
            "log": action.log,
            "tool": action.tool,
            "tool_input": action.tool_input,
            "trace_id": self.trace_id,
        })

    def on_agent_finish(self, finish: Any, **kwargs: Any) -> None:
        self.events.append({
            "timestamp": datetime.datetime.now().isoformat(),
            "type": "agent_finish",
            "output": finish.return_values["output"],
            "trace_id": self.trace_id,
        })

    def get_trace(self) -> Dict[str, Any]:
        return {"trace_id": self.trace_id, "events": self.events}

# Initialize LLM and Agent
llm = ChatOpenAI(model="gpt-4", temperature=0)
prompt = hub.pull("hwchase17/openai-tools-agent")
agent = create_openai_tools_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=False)

if __name__ == "__main__":
    trace_handler = MyTraceCallbackHandler()
    
    # Example 1: Product details query
    query1 = "What are the details for product PROD123?"
    print(f"\n--- Running query: '{query1}' ---")
    result1 = agent_executor.invoke({"input": query1}, config={"callbacks": [trace_handler]})
    print(f"Agent Response: {result1['output']}")
    with open(f"trace_{trace_handler.trace_id}.json", "w") as f:
        json.dump(trace_handler.get_trace(), f, indent=2)
    print(f"Trace saved to trace_{trace_handler.trace_id}.json")

    # Example 2: Order status query (new trace)
    trace_handler = MyTraceCallbackHandler() # Reset for new trace
    query2 = "What's the status of order ORDER456?"
    print(f"\n--- Running query: '{query2}' ---")
    result2 = agent_executor.invoke({"input": query2}, config={"callbacks": [trace_handler]})
    print(f"Agent Response: {result2['output']}")
    with open(f"trace_{trace_handler.trace_id}.json", "w") as f:
        json.dump(trace_handler.get_trace(), f, indent=2)
    print(f"Trace saved to trace_{trace_handler.trace_id}.json")

Step 2: Log Key Events and Data

Beyond the basic framework events, consider logging custom metrics or contextual information:

Persist these traces to a structured log file, a database, or a dedicated observability platform (like LangSmith, OpenTelemetry, or a custom solution).

By diligently capturing these traces, you gain invaluable visibility into your agent’s real-time behavior, paving the way for data-driven evaluation. The next step is to make sense of this raw trace data.

Building Actionable Evaluation Datasets from Traces

Raw traces are verbose. To effectively evaluate AI agents in production, we need to transform them into structured, queryable datasets. These datasets will serve as the foundation for analysis, metric calculation, and regression testing.

Step 1: Define Your Data Schema

Before extraction, determine what key pieces of information you need from each interaction for evaluation. A typical schema might include:

Step 2: Extract and Clean Trace Data

You’ll need a script to parse your saved trace files and extract relevant information into a tabular format, such as a CSV or a database table.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

# process_traces.py
import json
import pandas as pd
import glob

def process_single_trace(trace_file_path: str) -> dict:
    with open(trace_file_path, 'r') as f:
        trace_data = json.load(f)
    
    trace_id = trace_data['trace_id']
    events = trace_data['events']
    
    user_input = None
    agent_final_output = None
    llm_thoughts = []
    tool_calls = []
    total_latency_ms = 0

    llm_start_time = None
    
    for event in events:
        if event['type'] == 'llm_start':
            # Assuming the first LLM prompt often contains the user's initial query
            if not user_input and event['prompts']:
                # Heuristic: try to extract initial user prompt if available
                # This might need more sophisticated parsing for complex prompts
                user_input = event['prompts'][0] 
            llm_start_time = pd.to_datetime(event['timestamp'])
        elif event['type'] == 'llm_end' and llm_start_time:
            llm_end_time = pd.to_datetime(event['timestamp'])
            total_latency_ms += (llm_end_time - llm_start_time).total_seconds() * 1000
            llm_start_time = None # Reset for next LLM call
            llm_thoughts.append(event['response']) # Capture intermediate thoughts/responses
        elif event['type'] == 'tool_start':
            tool_calls.append({
                "tool_name": event['tool_name'],
                "tool_input": event['tool_input'],
                "tool_output": "PENDING" # Will be updated by tool_end
            })
        elif event['type'] == 'tool_end' and tool_calls:
            # Update the most recent tool call's output
            tool_calls[-1]["tool_output"] = event['tool_output']
        elif event['type'] == 'agent_finish':
            agent_final_output = event['output']
    
    # Simple heuristic to get user_input if not found earlier, assuming it's usually the first prompt for `invoke`
    if not user_input and events and events[0]['type'] == 'llm_start':
        user_input = events[0]['prompts'][0]

    return {
        "trace_id": trace_id,
        "timestamp": events[0]['timestamp'] if events else None,
        "user_input": user_input,
        "agent_final_output": agent_final_output,
        "tool_calls": json.dumps(tool_calls), # Store as JSON string
        "llm_chain_of_thought": "\n---\n".join(llm_thoughts),
        "total_latency_ms": round(total_latency_ms, 2)
    }

def create_evaluation_dataset(trace_dir: str = ".") -> pd.DataFrame:
    processed_traces = []
    for trace_file in glob.glob(os.path.join(trace_dir, "trace_*.json")):
        processed_traces.append(process_single_trace(trace_file))
    
    return pd.DataFrame(processed_traces)

if __name__ == "__main__":
    import os
    # Ensure some trace files exist from the previous step
    # Example usage: python main.py run the agent, then python process_traces.py
    
    # Create dummy trace files if they don't exist for demonstration
    if not glob.glob("trace_*.json"):
        print("No trace files found. Please run main.py first to generate traces.")
        # For demonstration, creating a mock trace file
        mock_trace_data = {
            "trace_id": "mock-trace-123",
            "events": [
                {"timestamp": "2026-08-28T10:00:00", "type": "llm_start", "prompts": ["What is the capital of France?"], "llm_model": "gpt-4"},
                {"timestamp": "2026-08-28T10:00:01", "type": "llm_end", "response": "Paris.", "trace_id": "mock-trace-123"},
                {"timestamp": "2026-08-28T10:00:01", "type": "agent_finish", "output": "Paris is the capital of France.", "trace_id": "mock-trace-123"}
            ]
        }
        with open("trace_mock-trace-123.json", "w") as f:
            json.dump(mock_trace_data, f, indent=2)


    df = create_evaluation_dataset()
    print("\n--- Generated Evaluation Dataset ---")
    print(df.head())
    df.to_csv("evaluation_dataset.csv", index=False)
    print("\nDataset saved to evaluation_dataset.csv")

Step 3: Enrich and Store the Dataset

Once you have the raw extracted data, you can enrich it:

Store this enhanced dataset in a suitable format (e.g., Parquet, PostgreSQL, specialized ML data store) for easy querying and historical analysis.

By building these comprehensive evaluation datasets, you move from anecdotal observations to quantitative insights, setting the stage for robust regression testing.

Implementing Automated Regression Tests for AI Agents

Automated regression tests are crucial for detecting performance degradations or unintended changes after agent updates, model fine-tuning, or tool modifications. This is the cornerstone for consistently evaluating AI agents in production.

Step 1: Curate Golden Datasets

A “golden dataset” is a collection of input-output pairs (or input-expected behavior paths) that represent desired agent performance for critical scenarios. These are your ground truth.

Example scenarios for an e-commerce agent:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# golden_test_data.yaml
- test_id: "prod_lookup_1"
  query: "What are the details for product PROD123?"
  expected_final_response_keywords: ["Product 123", "High-quality widget", "$29.99"]
  expected_tool_calls:
    - tool_name: "get_product_details"
      tool_input_regex: "PROD123"
  expected_no_tool_calls: []
  description: "Agent correctly retrieves details for a known product."

- test_id: "order_status_1"
  query: "What's the status of order ORDER456?"
  expected_final_response_keywords: ["Order 456", "Shipped", "ETA Aug 30"]
  expected_tool_calls:
    - tool_name: "check_order_status"
      tool_input_regex: "ORDER456"
  expected_no_tool_calls: []
  description: "Agent correctly checks status for a known order."

- test_id: "irrelevant_query"
  query: "Tell me a fun fact."
  expected_final_response_keywords: ["fun fact", "I can't", "I am an e-commerce assistant"]
  expected_tool_calls: []
  expected_no_tool_calls: ["get_product_details", "check_order_status"]
  description: "Agent handles irrelevant query gracefully without using domain tools."

Step 2: Define Evaluation Metrics

For each test case, define how you’ll measure success:

For subjective metrics, use techniques like LLM-as-a-Judge. A separate, powerful LLM (often a more capable, but slower, model like GPT-4-turbo) can evaluate an agent’s response against the golden expected response or a set of rubrics.

Step 3: Automate Test Execution

Integrate your golden dataset and metrics into an automated testing framework (e.g., pytest).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

# test_agent.py
import pytest
import yaml
import re
from main import agent_executor, MyTraceCallbackHandler # Assuming main.py from earlier

@pytest.fixture(scope="module")
def golden_tests():
    with open("golden_test_data.yaml", "r") as f:
        return yaml.safe_load(f)

def run_agent_and_get_trace(query: str):
    trace_handler = MyTraceCallbackHandler()
    result = agent_executor.invoke({"input": query}, config={"callbacks": [trace_handler]})
    return result['output'], trace_handler.get_trace()

@pytest.mark.parametrize("test_case", golden_tests())
def test_ai_agent_performance(test_case):
    print(f"\n--- Running test: {test_case['test_id']} ---")
    query = test_case['query']
    expected_response_keywords = test_case.get('expected_final_response_keywords', [])
    expected_tool_calls = test_case.get('expected_tool_calls', [])
    expected_no_tool_calls = test_case.get('expected_no_tool_calls', [])

    agent_output, trace = run_agent_and_get_trace(query)
    trace_events = trace['events']

    # Test 1: Final Response Keywords
    if expected_response_keywords:
        response_contains_keywords = all(
            keyword.lower() in agent_output.lower() for keyword in expected_response_keywords
        )
        assert response_contains_keywords, \
            f"[{test_case['test_id']}] Final response '{agent_output}' did not contain all expected keywords: {expected_response_keywords}"
    
    # Test 2: Tool Calls
    actual_tool_calls = [
        {"tool_name": e['tool_name'], "tool_input": e['tool_input']}
        for e in trace_events if e['type'] == 'tool_start'
    ]

    for expected_call in expected_tool_calls:
        found_match = False
        for actual_call in actual_tool_calls:
            if actual_call['tool_name'] == expected_call['tool_name'] and \
               re.search(expected_call['tool_input_regex'], actual_call['tool_input']):
                found_match = True
                break
        assert found_match, \
            f"[{test_case['test_id']}] Expected tool call '{expected_call}' not found in actual calls: {actual_tool_calls}"

    # Test 3: No Unwanted Tool Calls
    for unwanted_tool in expected_no_tool_calls:
        for actual_call in actual_tool_calls:
            assert actual_call['tool_name'] != unwanted_tool, \
                f"[{test_case['test_id']}] Unwanted tool '{unwanted_tool}' was called for query '{query}'"

    print(f"[{test_case['test_id']}] Test Passed.")

Run the tests:

1

pytest test_agent.py

This setup provides a powerful safety net. Before deploying a new version of your agent, run these regression tests. Failures indicate a regression that needs immediate attention.

Operationalizing Continuous Evaluation for AI Agent Reliability

Evaluating AI agents in production isn’t a one-time task; it’s a continuous process.

  1. CI/CD Integration: Incorporate your regression tests into your CI/CD pipeline. Any code changes should trigger these tests, preventing faulty agents from reaching production.
  2. Scheduled Production Data Runs: Periodically run your production agent against a fresh sample of real-world queries to detect drift. Use your evaluation datasets to compare current performance against historical benchmarks.
  3. Monitoring and Alerting: Set up dashboards to visualize key metrics (e.g., tool success rate, average latency, LLM-as-a-judge scores). Configure alerts for significant drops in performance or increases in error rates.
  4. Human-in-the-Loop Feedback: For complex or high-stakes scenarios, retain a human review process for a subset of agent interactions. Use this feedback to refine your golden datasets and improve evaluation metrics.
  5. A/B Testing: When introducing significant changes, consider A/B testing different agent versions in production to objectively measure impact on user engagement and business metrics.

By integrating these practices, you establish a resilient feedback loop that ensures your AI agents remain reliable, performant, and aligned with business goals even as they evolve.

Conclusion

Evaluating AI agents in production is a complex but critical endeavor. By systematically capturing execution traces, building robust evaluation datasets, and implementing automated regression tests, you can gain deep insights into agent behavior, detect regressions early, and build confidence in your AI systems. This hands-on approach empowers developers and MLOps teams to deploy and maintain intelligent agents with greater reliability and less risk, ensuring they consistently deliver value in real-world scenarios. Embrace these practices to elevate the quality and trustworthiness of your production AI agents.

FAQ

Further Reading

  1. LangChain Callbacks: https://python.langchain.com/docs/modules/callbacks/ - Official documentation on instrumenting LangChain applications.
  2. LangSmith for LLM Observability: https://docs.smith.langchain.com/ - A platform for debugging, testing, and monitoring LLM applications.
  3. Evaluating LLM Applications: https://www.deeplearning.ai/short-courses/evaluating-llms/ - A short course covering various evaluation techniques.

For more insights into MLOps best practices and AI engineering solutions, explore other articles on CodeCrux’s blog. Need expert assistance with your AI agent deployments? Check out our MLOps services.



Empower Your Business with Our Expert Solutions

Unlock the full potential of your projects with our professional services!

Get Started Today