Evaluate skill-equipped agents with Strands Evals and Amazon Bedrock AgentCore
A polished contract summary can still hide two very different failures: the agent used the wrong workflow, or it used the right workflow but skipped a crucial step. If your agents touch finance, HR, or compliance, you need measurements that separate those errors, and fast. Two complementary evaluation toolsets do exactly that: Strands Evals for replayable runs and deterministic checks, and Amazon Bedrock AgentCore Evaluations for trace-based, judge-based assessment of live or staged traffic.
Quick definitions
- Skill, a reusable package (often stored in SKILL.md following the Agent Skills standard) that contains Instructions, Tool bindings, Knowledge, Workflow, and Guardrails for a domain task.
- Router, the skill-selection component (model or rule) that decides which skill to invoke for a given user request.
- Judge, an LLM-based evaluator (a model used to score behavior) that grades whether the agent followed expected steps; judge outputs are useful but non-deterministic and require calibration.
- Invoked / loaded / executed, invoked means the runtime called the skill. Loaded means the skill assets were available to the runtime. Executed means the skill ran and returned output. The article uses these terms consistently where relevant.
Why modular skills still need targeted evaluation
Packaging business logic into skills makes agents cheaper and faster to specialize. Modularity does introduce two distinct failure modes you must measure separately:
- Routing (Skill Selection), the router picked an inappropriate skill for the task.
- Execution (Instruction Following), the right skill was invoked but steps were skipped, misapplied, or incompletely followed.
Evaluating only the final response leaves a blind spot. A fluent answer can hide either failure. Targeted evaluators give you actionable evidence: should you rewrite skill metadata and router rules, or fix step definitions, tool bindings, and guardrails inside the skill?
The three skill-focused evaluators you need
Two complementary systems provide the focused diagnostics teams need:
- Skill Selection Accuracy (SSA), available in both Strands Evals and AgentCore Evaluations. SSA returns a binary per-invocation pass/fail (reported as 1.00/0.00 and aggregatable to an accuracy percentage) to answer: “Was invoking this skill appropriate for the task?”
- Skill Instruction Following (SIF), available in both systems. A judge-based, five-level per-invocation rating answering: “How fully did the agent follow this skill’s prescribed steps?”
- SkillInvoked, Strands offers a deterministic SkillInvoked check that verifies whether a named skill was actually loaded or invoked. Equivalent deterministic checks may exist in other toolchains, but Strands provides this built-in today.
How Skill Instruction Following (SIF) is scored
SIF produces one of five labels, mapped to numeric scores:
- Fully Followed, 1.0
- Mostly Followed, 0.75
- Partially Followed, 0.5
- Minimally Followed, 0.25
- Not Followed, 0.0
By default, SIF counts as passing at Mostly Followed or better (0.75+). That threshold is a pragmatic starting point. Raise it for high-risk flows like finance and compliance, and consider lowering it for exploratory, research-focused agents.
A compact example and what it tells you
“SkillSelectionAccuracyEvaluator: score=1.00, pass=True”
“pdf-table-extraction: The skill directly matches the request.”
“SkillInstructionFollowingEvaluator: score=0.50, pass=False”
“pdf-table-extraction: The extraction phase was completed, the table-boundary phase was skipped.”
“Steps:
- Extract text with layout preservation: covered
- Locate table boundaries: skipped
- Summarize each table’s headline figure: partial
”
“SkillInvoked: score=1.00, pass=True”
“skill ‘pdf-table-extraction’ was invoked”
Interpretation: the router selected the right skill (SSA pass), the skill was loaded and invoked (SkillInvoked pass), but SIF failed because a critical step, locating table boundaries, was skipped. That points you to fixing the extraction workflow or adding guardrails inside the skill, not to retraining the router.
How to run these evaluators
Prerequisites:
- Python 3.10 or later for Strands SDK usage.
- An AWS account with Amazon Bedrock access and credentials with InvokeModel permission for the judge model (for AgentCore judge-based evaluation).
Install the tools (as shown in vendor docs):
- Strands SDK: pip install strands-agents-evals strands-agents
- AgentCore CLI: npm install -g @aws/agentcore
Input formats and integration signals (examples): Strands Evals ingests recorded agent runs as a Strands Evals Session or raw message lists. Examples of signals Strands recognizes include the Strands AgentSkills plugin, Claude Code, OpenAI Agents SDK, Codex, Gemini CLI, OpenHands, Google ADK, and direct SKILL.md reads. Some integrations may require adapters. AgentCore Evaluations operates on OpenTelemetry traces (spans) and can run on-demand, in batch, or continuously against production or staging traffic.
Example AgentCore CLI commands (from the docs):
- On-demand evaluation:
agentcore run eval –runtime <skill-runtime> –evaluator Builtin.SkillSelectionAccuracy Builtin.SkillInstructionFollowing –session-id <session-id> - Batch evaluation:
agentcore run batch-evaluation –runtime <skill-runtime> –evaluator Builtin.SkillSelectionAccuracy Builtin.SkillInstructionFollowing - Online (continuous sampling):
agentcore add online-eval –name HRSkillsProductionEval –runtime <skill-runtime> –evaluator Builtin.SkillSelectionAccuracy Builtin.SkillInstructionFollowing –sampling-rate 100 –enable-on-create
then: agentcore deploy
Strands Evals can be driven from recorded trajectories collected with a TracedHandler and exposes deterministic checks such as SkillInvoked for strict assertions. AgentCore judge-based results include spanContext metadata (sessionId, traceId, spanId), so each evaluation links back to the exact trace for debugging.
Traceability, auditability, and judge hygiene
Judge-based evaluators are powerful but non-deterministic. Persist the judge model name, prompt template, and raw judge output with each evaluation so you can audit decisions and detect drift. Each judge-based result includes spanContext metadata to locate the originating trace and inspect tool calls, inputs, and model outputs, which is crucial for root cause analysis.
Concrete triage and remediation checklist
- If SSA is failing often for a skill:
- Check router logs and intent and confidence scores. Add monitoring on router confidence.
- Tune the intent classifier: add labeled examples, negative samples, and synonyms or aliases for the skill.
- Enrich skill metadata (aliases, intent phrases) or add a short disambiguation step in the router.
- If SIF falls below Mostly Followed:
- Open the trace for the failed run (spanContext) and inspect which steps were skipped or partial.
- Add intermediate assertions inside the skill (post-condition checks, format validators) so missing steps fail fast.
- Break complex skills into sub-skills with clear outputs, and add unit tests that mock tool responses.
- Increase the agent’s context or provide worked examples in SKILL.md to clarify expected step sequencing.
- If SkillInvoked assertions fail (deterministic):
- Fail fast in CI. A deterministic check means the skill wasn’t loaded or the invocation path is broken.
- Check packaging, import paths, runtime config, and deployment artifacts. Add pre-deploy smoke tests that call the skill runtime.
Practical recommendations for adoption
- Run both SSA and SIF. They answer different questions, separating routing from execution.
- Start with built-ins before custom logic. Use built-in evaluators to set a baseline, then add tool-level templates or custom rules where the judge lacks domain context.
- Use deterministic checks for hard gates. SkillInvoked-type assertions should block deploys for non-negotiable routing requirements. Use judge evaluations for softer gates and diagnostics.
- Integrate into CI/CD, but be cautious. Treat failing evaluations like failing unit tests, with the caveat that judge-based checks can be flaky. Gate production with deterministic checks and use judge results for deeper quality control.
- Sample judge evaluation in production. Start small, single-digit percent sampling for low-risk flows, and increase to 100% only for regulatory or high-risk paths to manage cost and latency.
- Calibrate per surface. Judge prompts, model versions, and trace formats differ. Tune thresholds separately for Strands and AgentCore before enforcing them in CI/CD.
- Pair skill-level evaluation with end-to-end tests and business KPIs. A passing SIF does not guarantee the intended business outcome. Keep scenario-level checks and outcome metrics in place.
Known limits and planning points
- Judge drift and calibration: expect to recalibrate when judge models, prompts, or evaluation rubrics change.
- Cost and latency: continuous judge evaluation consumes model invocations. Use sampling for production traffic and prioritize critical flows for full evaluation.
- Privacy and compliance: traces and recorded trajectories may contain PII. Add redaction and data governance before exporting traces for evaluation.
- Versioning SKILL.md: keep skills versioned and record which skill version produced each run so historical evaluations remain meaningful.
- Telemetry gaps: in mixed-harness environments, skill extraction may miss invocations. Verify signal coverage and use deterministic checks where signals are unreliable.
Mini case: payroll posting gone wrong (and how evaluators help)
A payroll agent posts deductions but one employee’s tax withholding is mis-classified because a table extraction step dropped a header row. SIF shows “Partially Followed” for the payroll-extraction skill and pinpoints the skipped table-boundary step. SSA for the chosen payroll skill is green, and SkillInvoked confirms the right skill ran. The fix is clear: add a table-boundary guardrail and unit tests in the skill, not a router retrain. Without SIF and SkillInvoked you might chase the router or manually inspect many runs, expensive and slow.
Key takeaways, quick questions you should be able to answer
- What does Skill Selection Accuracy tell me?
It answers whether invoking a particular skill was appropriate for the user’s request. It’s a binary yes/no evaluator available in both Strands Evals and AgentCore Evaluations (reported per invocation as 1.00/0.00).
- How does Skill Instruction Following measure success?
SIF rates each invoked skill on five levels (Fully to Not Followed) mapped to numeric scores (1.0 to 0.0). “Mostly Followed” (0.75) or better is the default pass threshold; tune that per risk surface.
- When should I use SkillInvoked?
Use it for deterministic checks in CI or production sampling when you have non-negotiable routing requirements; it verifies whether a named skill was actually loaded/invoked. Strands includes this check today.
- Can I run these checks on live traffic?
Yes. AgentCore Evaluations runs judge-based evaluators on OpenTelemetry traces on-demand, in batch, or continuously (with sampling). Strands is designed for recorded runs and CI-driven workflows.
- How do I trace a judge result back to the original run?
Each judge-based evaluation includes spanContext metadata (sessionId, traceId, spanId) so you can open the exact trace and inspect inputs, tool calls, and model responses for root-cause analysis.
“Skills make agents inexpensive to specialize, but a plausible final answer does not prove that the agent selected the right procedure or followed it.”
This is intentional. Use SSA to ask “did we pick the right tool?”, use SIF to ask “did the tool do everything it was supposed to?”, and add deterministic checks for hard guarantees. Instrument evaluation into your CI/CD and production monitoring, persist judge artifacts for auditability, and you’ll move from reactive forensic investigations to proactive quality control, the difference between automation you can trust and automation that looks right until it breaks you.