Build reliable AI agents with Amazon Bedrock AgentCore Evaluations

• TL;DR — Business impact
• Replace one-off demos with repeatable, auditable agent evaluation so AI automation delivers predictable results for customers and stakeholders.
• Move from bespoke testing pipelines to a managed evaluation layer that plugs into CI/CD, production monitoring, and governance.
• Detect failures earlier (wrong tool choice, bad API parameters, hallucinated synthesis) and tie fixes to measurable metrics like Goal Success Rate and Tool Selection Accuracy.

Your demo impressed execs. Two weeks later a customer hits a hallucination, the agent calls the wrong API, and you’re scrambling to reproduce the issue. A single transcript tells you what happened once — not what typically happens. That’s the reliability gap AgentCore Evaluations aims to close.

What AgentCore Evaluations does for AI agents

Amazon Bedrock AgentCore Evaluations is a managed service that automates end-to-end agent evaluation. It collects extended OpenTelemetry traces (prompts, tool calls, model outputs, and parameters) so you can trace what the agent actually did. Those traces feed a set of evaluators that score agent behavior across three levels: the entire session (the full conversation), the trace (a single exchange or round trip), and the span (an individual operation or tool call).

Put simply: instead of spot-checking transcripts, you get continuous, measurable signals about planning, tool usage, and response synthesis. That’s the difference between reacting to surprises and preventing them.

How it works (plain-English)

• OpenTelemetry traces — OpenTelemetry (OTEL) is a tracing standard; AgentCore extends it with generative-AI conventions so traces capture prompts, tool invocations, LLM completions, and model settings.
• Three scoring layers — Session (goal success over a conversation), Trace (one request/response exchange), Span (a single tool call or sub-operation).
• Evaluator engines — Built-in evaluators, LLM-as-a-Judge prompts, ground-truth comparison, and code-based evaluators (AWS Lambda) provide flexible scoring options.
• Integration — Results are written to AWS CloudWatch and surfaced in the AgentCore Observability dashboard for trends, drill-downs, and alerts.

Evaluation approaches and built‑in metrics

AgentCore supports three main evaluator types:

• LLM-as-a-Judge — use a model to reason about quality and give human-like judgments (useful for nuance and subtle heuristics but variable and costlier).
• Ground-truth comparison — compare outputs to expected_response, expected_trajectory, or assertions for deterministic scoring of correctness and goal achievement.
• Code evaluators (AWS Lambda) — deterministic, cheap at scale, and ideal for format checks, numeric accuracy, and business-rule enforcement.

There are 13 built-in evaluators that cover common signals: Goal Success Rate (session), Helpfulness, Correctness, Coherence, Faithfulness, Harmfulness, Instruction Following, Response Relevance, Context Relevance, Refusal, Stereotyping (trace level), and Tool Selection / Parameter Accuracy (span level).

“Agent behavior in production often differs from demos because LLMs are non‑deterministic — a single run shows what can happen, not what typically happens.”

That non-determinism is why the service recommends repeated trials (roughly 10 runs per input as a starting point) to measure variance and build statistical confidence rather than trusting single runs.

On‑demand evaluation vs online evaluation

Two operational modes cover development and production needs:

• On‑demand evaluation — Run during development, CI/CD, and regression testing. Use the Evaluation Client or the On‑Demand Evaluation Dataset Runner to attach the same tests you’ll use in production to your pipeline.
• Online evaluation — Continuous sampling and scoring of production traffic. Outputs stream to CloudWatch and the AgentCore Observability dashboard so you can alarm on regressions and track drift.

The on‑demand API returns up to 10 evaluation results per call (useful for batch runs and CI). Online evaluation supports sampling strategies so you can monitor representative production flows without scoring every single session with an LLM judge.

Real-world scenarios (mini case studies)

Sales automation agent: a bot books demos by calling CRM and calendar APIs. A failure could be a wrong calendar ID or incorrect timezone. Instrumentation captures the tool call span. A Tool Selection / Parameter Accuracy evaluator would flag the bad API parameters, while a session-level Goal Success Rate evaluator detects whether the booking actually completed.

Customer support for regulated workflows: an agent must refuse disallowed requests and provide evidence when it redirects to a human. Use a mix of LLM judges to assess refusal appropriateness and Lambda evaluators to validate that the agent attached required audit metadata and compliant phrasing. Ground-truth assertions verify mandatory regulatory fields.

Best practices for productionizing agent evaluation

• Instrument first — capture prompts, tool calls, model parameters, user metadata, and expected responses so evaluators have complete context.
• Use multi-dimensional metrics — session, trace, and span scores help pinpoint whether the issue is planning, a tool call, or synthesis.
• Calibrate judges with SMEs — have subject matter experts review LLM-judge prompts and sample outputs to align scoring with business expectations.
• Mix evaluators — combine LLM judges for nuance and Lambda code evaluators for deterministic, low-cost checks.
• Gate releases — tie the same evaluators into CI/CD so deployments fail fast when critical metrics regress (e.g., core flow Goal Success Rate drops).
• Monitor drift — set alerts for meaningful shifts (illustrative guidance: investigate if Goal Success Rate drops >5% week-over-week).

CI/CD checklist (practical)

• Instrument flows with OTEL generative-AI conventions.
• Create a small baseline dataset of representative sessions and expected outcomes.
• Run on‑demand evaluations in your pipeline; fail the build if core evaluators cross thresholds.
• Use Lambda evaluators for costly per-session checks and judge evaluators for sampling and nuance.
• Push evaluation outputs to CloudWatch and tie them to dashboards and alarms.

Limitations, tradeoffs, and compliance concerns

LLM judges provide human-like reasoning but inherit model variability and cost. Lambda evaluators are deterministic and cheaper but require engineering effort to encode rules. Cross‑Region inference for built-in judges improves availability but can raise data-residency or compliance concerns—AgentCore supports custom evaluators and single-Region configurations if you need to keep data within a jurisdiction.

Also, evaluation focuses on correctness, coherence, and behavior quality. Latency and cost remain important operational metrics that standard observability tools should continue to monitor in parallel.

“Evaluating agents must measure the entire interaction flow: tool selection, parameter correctness, and final response synthesis.”

Who should care

• Product leaders — want reliable, measurable outcomes and defensible KPIs for AI features.
• ML engineers — need repeatable test suites, judge calibration, and trace instrumentation.
• SRE/ops — require continuous monitoring, alarms, and dashboards tied to production traffic.
• Compliance and risk teams — need auditable trails and deterministic checks for regulated flows.
• Sales ops & customer success — want assurance that automation doesn’t damage customer relationships or data integrity.

Quick start checklist

• Instrument agents with OpenTelemetry generative-AI conventions.
• Choose 3 starter evaluators: Goal Success Rate (session), Tool Selection Accuracy (span), and Correctness or Helpfulness (trace).
• Run on‑demand evaluations during CI with a baseline dataset; require statistical confidence via repeated trials (~10 runs per input to start).
• Deploy online sampling for production and route evaluation outputs to CloudWatch and AgentCore Observability.
• Use Lambda evaluators to enforce deterministic business rules and keep costs predictable at scale.

Key takeaways & questions

• How does AgentCore change agent testing?
  

  It standardizes end-to-end agent evaluation by consuming extended OTEL traces and scoring at session, trace, and span levels so you can integrate evaluations into CI/CD and production monitoring rather than relying on ad-hoc transcript checks.

• When should I use LLM judges vs code (Lambda) evaluators?
  

  Use LLM-as-a-Judge for nuanced, human-style judgments where context and trade-offs matter; use Lambda/code evaluators for deterministic, high-volume checks like format validation, numeric accuracy, and business-rule enforcement.

• How many runs do I need to trust a score?
  

  Because LLMs are non-deterministic, run repeated trials (around ten per input as a starting guideline) to estimate variance and avoid decisions based on single runs.

• Can I use the same evaluators in CI/CD and production?
  

  Yes — tying the same evaluator configurations to pre-deploy gates and online monitoring ensures consistent quality standards across development and production.

Next steps

Start small: instrument a core flow, add three evaluators (session Goal Success, trace Correctness, span Tool Selection), run on‑demand tests in your pipeline, and enable online sampling to watch for drift. Mix LLM-based judgment for nuanced cases with Lambda evaluators for deterministic checks, and calibrate judge prompts with SMEs.

For teams ready to explore, the bedrock-agentcore GitHub repo has sample code and the AWS docs describe API details and configuration options. Treat evaluation as part of your product definition — the same way you accept build and test automation, treat continuous evaluation as a production discipline for AI agents.

Call to action: pick one agentic flow to instrument this quarter. Add evaluators as CI gates, enable online sampling, and set an alert threshold so the first time Goal Success Rate moves meaningfully you’ll get notified before customers do.

Authors and contributors behind this capability include Akarsha Sehwag, Ishan Singh, Bharathi Srinivasan, Jack Gordley, Samaneh Aminikhanghahi, and Osman Santos.