AWS Releases Agent-EvalKit, an Open-Source Tool for Testing AI Agents

Amazon Web Services has released Agent-EvalKit, an open-source toolkit for systematically testing AI agents. Instead of judging only a model's final answer, it tracks the agent's entire execution path: tool calls, intermediate states, and the flow of data between steps.

The project addresses a growing problem for companies deploying AI agents: standard unit tests and end-answer comparisons don't catch errors that occur along the way, when an agent independently selects tools and combines results from several sources. Classic testing checks whether an answer is correct, but it doesn't reveal why the agent went wrong or at which step.

How the tool works

Agent-EvalKit walks the user through six phases, each run as a command inside a coding assistant: planning the evaluation strategy, generating test cases, adding OpenTelemetry-compliant trace instrumentation, running the agent and collecting execution traces, implementing metrics, and generating a report with recommendations tied to specific pieces of code.

The tool integrates directly with coding assistants, including Claude Code, Kiro CLI, and Kilo Code, so the entire evaluation process happens in the same environment where the agent's code is written. Instead of a separate testing framework, developers get a set of slash commands that guide them step by step from a test plan to a finished report.

Three dimensions of evaluation

The key difference from simpler tools is that Agent-EvalKit evaluates three dimensions of agent behavior at once: faithfulness, meaning whether the answer is actually grounded in the data returned by tools; the accuracy of the parameters passed to called functions; and overall response quality. Separating these three metrics makes it possible to pinpoint the stage in the agent's chain of actions where it starts generating content unrelated to real data.

Agents that independently choose tools and sequence operations across multiple sources produce behaviors that outcome-only testing cannot fully characterize - Agent-EvalKit documentation, AWS

In a case study published by AWS, a travel-planning research agent scored 83.9 percent on overall response quality and 64.5 percent on tool-parameter accuracy, but only 32.3 percent on faithfulness. The gap revealed systematic hallucinations: when a web search returned no results, the agent still generated specific, plausible-sounding answers instead of reporting that no data was available.

Market context

The release of Agent-EvalKit fits a broader trend in which cloud and agent-framework providers are investing in testing and observability tools for agents, as customers increasingly report that production agents fail in ways that are hard to diagnose with standard methods. AWS is developing the Strands Agents SDK, an open-source framework for building agents, alongside it, and Agent-EvalKit is a natural extension of that tooling stack.

The GitHub repository is released under the Apache 2.0 license, allowing unrestricted commercial use and modification of the code. The project requires Python 3.11 or newer and the uv package manager, and installation comes down to a single command that pulls the latest version from the repository's main branch.

Relevance for Polish teams

For Polish companies deploying AI agents in products or internal processes, a tool like this addresses an issue that comes up increasingly often in conversations with enterprise clients: an agent looks like it's working correctly in a demo, but in production it makes decisions based on incomplete or empty tool data, something that's invisible without tracing the full execution path. An open, free toolkit that integrates with popular coding assistants lowers the barrier to entry for teams that don't have budget for commercial agent-observability platforms.

Agent-EvalKit doesn't replace human verification, but it automates the most labor-intensive part of the process: generating test cases and pinpointing which pieces of code are responsible for specific errors. For teams working with the LangGraph or CrewAI frameworks, which AWS lists as supported, this means the tool can be connected to existing code without rewriting the agent for a new testing framework.