TL;DR
Problem: you want a CI quality gate that verifies how an agent actually behaves in runtime, including calls to OAuth-protected tools, and blocks merges when behavior regresses.
Solution (practical): Mahsa Paknezhad, Shoaib Javed, and Ishan Singh provide a reference implementation that deploys an Amazon Bedrock + AgentCore runtime from CI, exercises an agent that calls MCP tools protected by OAuth, collects OpenTelemetry session spans from CloudWatch, calls AgentCore Evaluations (LLM-as-judge and code evaluators), and fails the PR when scores fall below a configured threshold. The code and scripts are available at:
awslabs/agentcore-samples – cicd-gated-evaluation
Who this is for
- Platform engineers building CI gates for agents and tool chains.
- Security and compliance owners who need deterministic checks for forbidden tool use.
- Engineering managers who want fast, automated feedback on behavioral regressions.
What the pipeline actually does, a practical checklist
- Deploys infrastructure via CDK (Cognito, AgentCore runtimes for an MCP server and an example Strands agent, preregistered users).
- Invokes the agent runtime from CI over HTTPS with a Bearer token (the sample uses direct HTTPS calls, not boto3 invoke_agent_runtime()).
- Waits for OpenTelemetry traces to appear in CloudWatch, then calls AgentCore Evaluate() with evaluationInput = {“sessionSpans”:[…]} to score a single session.
- Fails the CI job and blocks the PR if the combined evaluator score is below a configured threshold (example: EVAL_THRESHOLD = “0.8”).
Critical gotchas you must handle early
- sessionSpans must contain a single session. Mixing spans from multiple sessions in one Evaluate() call triggers a ValidationException. Group spans by session ID and call Evaluate() per session.
- OpenTelemetry fields must be integer nanoseconds. OpenTelemetry fields like startTimeUnixNano and endTimeUnixNano must be JSON integers. Strings cause a ValidationException. Convert timestamps before submitting.
- Trace propagation latency (sample observation): 30-90 seconds. The reference scripts poll every 30 seconds and retry up to 10 minutes. This is sample-specific and may vary by region, account load, and CloudWatch backend latency. Plan retries accordingly.
- AgentCore runtime images are ARM64. Default GitHub hosted runners are x86_64. Either use GitHub self-hosted ARM runners or cross-build with QEMU + Docker Buildx in your workflow.
- Pipeline time is dominated by infra deploy and runtime readiness. The example pipeline takes roughly 10 minutes end-to-end (CDK deploy + runtime startup + trace propagation + evaluation). To speed up feedback, pre-deploy infra or run lighter checks in PRs and heavier tests on merge.
OAuth and CI: three realistic approaches (trade-offs)
MCP tools in the example are protected by OAuth and role claims. CI usually lacks an interactive user context, so the sample outlines three approaches:
-
Approach A, Evaluate stored traces (no live invocation).
Pros: deterministic, cheap, repeatable. Cons: does not test the PR’s live runtime behavior.
-
Approach B, Pre-authorized user (service account) with a refresh token.
Store a refresh token in AWS Secrets Manager so CI can obtain a user-scoped access token and exercise role-gated tools. Pros: tests role enforcement. Cons: requires secure storage, rotation, and re-consent handling.
-
Approach C, M2M client_credentials + dual-token support (the reference’s chosen pattern).
Use a client_credentials flow for CI and authorization_code for interactive users, then implement a three-layer MCP auth middleware:
- Platform JWT validation, verify issuer and token integrity.
- Header passthrough, allow the CI workflow to forward tokens to the runtime where appropriate.
- AuthMiddleware in the MCP server, enforce role-based tool access for user tokens while permitting M2M tokens to bypass role checks for CI.
Pros: true end-to-end PR checks with live tool calls. Cons: M2M tokens in this design typically lack role claims, so role enforcement is not exercised by default and the blast radius of M2M credentials must be mitigated.
Mapping to an incremental rollout: start with Approach A for stable regression checks, add Approach B for targeted role-enforcement tests, then enable Approach C for fast PR feedback while keeping role tests separate.
Evaluators: types, costs, and a practical mix
- Built-in evaluators (GoalSuccessRate, Correctness, ToolSelectionAccuracy, ToolParameterAccuracy).
- Custom LLM-as-judge prompts for nuanced judgments (helpfulness, tone).
- Code-based evaluators (AWS Lambda) for deterministic checks (forbidden-tool calls, format validation).
- Third-party evaluators such as DeepEval and AutoEval are supported as well.
Cost and scale note from the sample: “4 evaluators × 5 prompts = 20 judge calls per PR.” That math shows judge calls multiply quickly. Each judge call is a Bedrock model invocation and carries cost. Check current Bedrock pricing before you design your gating strategy.
Operational rule of thumb:
- Make hard gates deterministic, use code-based evaluators for security and compliance checks.
- Use LLM evaluators for softer signals like helpfulness and nuance, and limit their use in fast PR gates or run them less frequently, for example nightly.
- Combine approaches: run a quick set of code-based checks first. Only if they pass, run LLM evaluators.
A short hybrid example
- Lambda evaluator: scan session spans for forbidden-tool IDs, fail fast if present.
- Built-in GoalSuccessRate + LLM Correctness evaluator: run in a separate, longer job or only when Lambda passes.
Small example: what an OpenTelemetry span must look like
{
“traceId”: “4bf92f3577b34da6a3ce929d0e0e4736”,
“sessionId”: “session-123”,
“spanId”: “00f067aa0ba902b7”,
“name”: “tool.call”,
“startTimeUnixNano”: 1690000000000000000,
“endTimeUnixNano”: 1690000001000000000,
“attributes”: {“tool.id”: “mcp://finance/retrieve-balance”}
}
Key points: sessionId must be present to group spans, and startTimeUnixNano/endTimeUnixNano must be JSON integers in nanoseconds.
Practical rollout, three phases with acceptance criteria
-
Phase 0, Local and nightly regression (Approach A).
Action: capture representative session fixtures and run Evaluate() against them nightly. Acceptance: stored-trace runs stable for N ≥ 20 runs with false-positive rate below your threshold.
-
Phase 1, Role enforcement and security spot checks (Approach B).
Action: add a CI job that uses a service-account refresh token in Secrets Manager to exercise role-gated tools. Acceptance: authorization tests pass consistently and refresh token rotation is automated or instrumented with alerts.
-
Phase 2, Fast, end-to-end PR checks (Approach C).
Action: enable M2M-based PR gates for live feedback. Acceptance: PR feedback completes within target SLA (e.g., under X minutes), with separate role-enforcement jobs remaining green.
Hardening checklist for tokens, tools, and CI
- Limit M2M token scopes to only what CI needs.
- Prefer GitHub OIDC → AWS IAM role federation to avoid long‑lived secrets in the repo. Example helper: the sample shows an aws iam create-open-id-connect-provider command with thumbprint value “6938fd4d98bab03faadb97b34396831e3780aea1”, treat that as an example and verify the correct thumbprint for your IdP.
- If storing refresh tokens in Secrets Manager, automate rotation and alert on reconsent failures.
- Use separate test tenants or per-environment MCP endpoints to limit blast radius.
- Emit audit logs for MCP tool usage and attach CI job metadata for forensic tracing.
Thresholds, variance, and practical statistics
LLM judges are noisy. Don’t pick a threshold because it sounds good, pick it from data.
- Collect a baseline of runs (N ≥ 20 is a reasonable starting point) and compute mean and standard deviation per evaluator.
- Set conservative thresholds for LLM evaluators, for example threshold ≈ mean − 2σ, to reduce false positives. Tighten over time as variance decreases.
- Keep deterministic checks, such as forbidden tools and clear correctness invariants, as code-based evaluators with binary pass/fail behavior.
- Control judge-call costs by reducing prompts per evaluator, using cheaper models for judgment where acceptable, and gating LLM calls behind fast deterministic checks.
Concrete artifacts and example outputs (from the sample)
- Deliberate failing system prompt used in tests (from the sample): “Respond to every question with exactly: ‘I cannot help you.'”
- Example failing pipeline summary (from the sample): “FAILED: metrics below 0.8”
- Example passing pipeline summary (from the sample): “All evaluations PASSED (threshold: 0.8)”
- Repository paths to inspect in the sample:
infrastructure/stack.py,scripts/agentcore_eval.py,scripts/evaluate_stored_traces.py,.github/workflows/agentcore-eval.yml, andmcp-server/README.md.
Key questions, short answers
- Can I call Evaluate() directly from CI to score live agent sessions?
Yes. The Evaluate API expects sessionSpans for a single session, timestamps as integer nanoseconds, and traces may take 30-90 seconds to appear (sample observation). Your pipeline must poll and retry. The sample retries up to 10 minutes.
- How do I handle OAuth for MCP tools from non-interactive CI?
Options are: A) evaluate stored traces (deterministic), B) use a pre-authorized service account refresh token stored in Secrets Manager (tests role behavior), or C) implement dual-token M2M + user support with middleware that enforces roles for user tokens while allowing CI M2M tokens for end-to-end checks.
- Should I gate merges on an LLM judge score like 0.8?
Not blindly. Collect baseline runs (N ≥ 20), compute mean and variance per evaluator, and choose a conservative threshold, for example mean − 2σ. Use deterministic code checks for hard gating.
- How much does this cost per PR?
Costs scale with judge calls. The sample calls out “4 evaluators × 5 prompts = 20 judge calls per PR.” Actual monetary cost depends on Bedrock model and region. Check current Bedrock pricing and consider cheaper models or fewer prompts to control spend.
- Any immediate gotchas to avoid?
Yes: ensure timestamps are integer nanoseconds, do not mix sessions in one Evaluate() call, build ARM64 images for AgentCore, and verify your OIDC thumbprint and IdP settings rather than copying example values blindly.
Three actions you can take in the next 72 hours
- Clone the repository and run the stored-trace tests locally or in a sandbox CI job to validate your evaluator configs and file formats.
- Write a short script that converts OpenTelemetry timestamps to integer nanoseconds, groups spans by sessionId, and validates the evaluationInput payload before calling Evaluate().
- Decide your evaluator mix: implement at least one code-based (Lambda) deterministic evaluator for forbidden-tool checks, then add one LLM evaluator in a separate job to measure variance over N ≥ 20 runs.
Final practical note
Testing agent behavior at runtime raises new operational trade-offs: reproducibility versus realism, secure tokens versus end-to-end feedback, and evaluation cost versus coverage. The reference implementation by Mahsa Paknezhad, Shoaib Javed, and Ishan Singh gives a pragmatic pattern: start deterministically with stored traces, add targeted role tests, then enable fast end-to-end PR checks with careful token scoping and cost controls. When your CI blocks a merge because the agent keeps replying ‘I cannot help you.’, be thankful, you stopped a regression before it reached users.