Before any AWS field seller put a live question to the assistant, a pre-launch evaluation of the Customer Insights Agent caught a number that never reached a Weekly Business Review. A customer’s three-month revenue on a service was $410.08. The agent, fluent and certain, reported “$410.08 Million.”
That is off by a factor of a million: a four-hundred-million-dollar figure for a four-hundred-dollar fact. Six orders of magnitude, not a rounding error.
It was not a lonely typo. In the same pre-launch checks, Sales Revenue was labeled as Sales Revenue excluding Adjustments. An annual recurring revenue estimate appeared that was not in the data. The agent offered its own arithmetic as if a source system had handed it over.
The monitors stayed green. AWS’s Sales, Marketing, and Global Services (SMGS) Operations team later wrote that those checks told them the system had responded, not whether the answer was right. The evaluation is what stopped the number. Uptime would not have.
When the dashboard stays green
Traditional software fails loudly. A null pointer, a 500, a job that never finishes: someone gets paged. Agents fail softly. They can take a different path for the same input, call the wrong tool, invent a metric, and still sound like the sharpest analyst in the room.
The most damaging failure is not a crash. It’s a fluent, confident answer that is quietly wrong.
That is the SMGS team’s line, from its engineering write-up on running this assistant in production. AWS has illustrated the same pattern on a separate demonstration airline-reservation agent (empty replies and bad handoffs that need not raise the error rate), but those are illustrations from a different system, not measurements from field sales. Infrastructure health is not answer quality.
The assistant lets sellers and sales leaders ask business questions in plain English through Amazon Quick, with answers scoped by territory and role. A Performance Insights Agent writes narratives from sales data. A Customer Insights Agent pulls together product usage, support, contracts, and engagement. A Data Insights Agent turns plain English into SQL over structured analytics. Dashboards sit beside the chat. The system supports front-line sellers and leadership cadences, including Weekly Business Reviews, for AWS Field Sales worldwide.
Useful. Also a machine that can misstate the business in the voice of the business. For a sales team, that confusion costs a decision, not a ticket.
Name the miss before you measure it
The team treats evaluation as five jobs at once: safety, quality, performance, cost, and operational control. Those jobs apply at three levels: the tool, the individual agent, and the multi-agent handoff. A specialist can look sharp while the router sent the question to the wrong one. Tool success, answer quality, and orchestration quality are different contracts.
The platform’s own scoring hierarchy does not line up with that shape. AgentCore Evaluations scores sessions, traces, and individual tool calls. A session can look successful while a handoff was wrong. If you only adopt the platform’s labels, you can miss the failure the sales team actually fears.
On accuracy, they use a hard rule. Any one of six failures marks the response wrong:
- Metric swap: the right shape of number, the wrong measure.
- Wrong period: last quarter’s truth delivered as this quarter’s.
- Wrong entity: a clean answer about a different customer, account, or seller.
- Pure hallucination: a figure or fact with no source.
- Wrong symbols: units, tickers, or labels that change the meaning.
- Unauthorized calculation: a derived number the source data does not sanction.
They track those in concrete units: occurrences per 100 responses, the share of answers that cite the correct time window, entity-resolution accuracy, grounding rate, and scope adherence. Grounding rate, in their usage, is the share of reported figures that can be traced to retrieved source material.
That is not the same check as the built-in faithfulness judge in Amazon Bedrock AgentCore Evaluations, which scores whether an answer stays consistent with conversation history and tool outputs. A third check, context relevance, asks whether the retrieved material even contained what the question needed. Mix those up and you will “fix retrieval” when the generator is inventing, or the reverse.
Where there is one right revenue figure, a deterministic check against ground truth is the right instrument (renewal date, contract status, entity, SQL result). Where the question is a judgment, the team uses an LLM-as-a-judge rubric. As a managed AgentCore method, the judge returns a label, a score, and an explanation. It is not a vibe. Read the judge prompt before you treat the score as your business definition. A faithfulness judge that means “consistent with tool output” will not catch a unit error a finance partner would reject in one glance.
The six sales failures are not built-in labels you can switch on. The team says its stack combines AgentCore’s built-in evaluators, six customized domain evaluators, managed evaluators from DeepEval and AutoEval, and those deterministic checks. Custom evaluators can run as code in AWS Lambda. You have to write the definition of a wrong dollar.
All of that hangs on a golden set: a living, version-controlled collection of quality, safety, latency, and cost cases. Theirs covers four question types (comprehensive, reporting, reasoning, and ambiguous), drawn from production queries, caught regressions, and synthetic edge cases. Topic coverage is revisited monthly. The write-up does not say how large the set is. What you can check is whether last month’s embarrassing miss is still in it.
A smoke test is not a release
Upstream data pipelines refresh every week. Quality can decay on a Tuesday with no code shipped. In this kind of automation, the warehouse refresh is a release. So is a prompt tweak. So is a seller typing a ticker instead of a legal name.
They learned the coordination version of that in pre-production, before the sentinel even enters the story. A mismatch among a data change, a tool change, and the prompt update that was supposed to accompany them dropped the pass rate from 78 percent to 38 percent. Three reasonable edits, one bad morning.
Every staging deployment now runs a sentinel set of 15 high-risk cases, not the full golden set. The sentinel is reviewed monthly. Regressions that reach production get promoted into it. The team is blunt about the limit: clearing the sentinel is necessary, and it is not sufficient. Production covers tens of thousands of query patterns across hundreds of account types, geographies, and seller segments. Fifteen cases are a smoke alarm. They are not a fire inspection. That limit is about coverage, not about the 78-to-38 morning.
Clearing the sentinel is necessary. It is not sufficient.
The pre-release quality gate runs the full candidate end to end against 200 held-out questions. A candidate fails automatically if its aggregate score regresses against its own prior run, if it fails to beat the production baseline, or if more than a defined share of those 200 questions falls below a per-question bar. That share is not published. If you copy the pattern, you still have to set the bar yourself.
They also score reasoning against frozen, replayed tool responses, so a flaky upstream call does not masquerade as a reasoning win or a reasoning loss. Replaying a known tool result is how you know what you measured.
The 2 a.m. problem
Three clocks run after release, and they are not the same measurement.
Canary evaluation uses 30-minute windows. The team found 15-minute cycles too noisy, and 60-minute cycles slow enough that two hours could pass before a degradation registered. That is their operating choice, not a documented platform default. On-call paging waits for three consecutive windows over the threshold, so a sustained breach can page within 90 minutes. That clock is an alarm design.
Online evaluation samples 10 percent of live traffic and scores it in the background. AgentCore supports a configurable sample. 10 percent is the rate this team chose. That clock is a sample, not a page.
A round-robin through the golden set covers it in 24 hours, and a full golden-set run executes once a day at the gate thresholds. That clock is a sweep. It is not how fast someone gets woken up.
The team reports that before this observability stage, mean time to detect was days to weeks, and that it is now under four hours. Under four hours is also the target on their metrics table. Treat that figure as their claim and their bar, not as a stopwatch reading you can reconcile with the 90-minute page or the daily sweep. Other targets on that table (including guardrail latency and an automation detection rate) are targets, not reported results, and they do not belong in the same sentence as a detection story.
In one live drift window, customer-identification quality moved from 98 percent at 1:30 a.m. to 81 percent at 2:00 a.m. and 80 percent at 2:30 a.m. The cause was not a bad deploy. Live queries used stock tickers, shorthand names, and typos, including “what’s going on with AMZN?” The question the setup is built to answer is specific: what happened to customer identification accuracy between 1 a.m. and 3 a.m.?
Safety is not parked in one of those windows. As the team runs them, Amazon Bedrock Guardrails execute synchronously on 100 percent of traffic, on the way in and on the way out. The sales-relevant risk is a pasted CRM note or customer email that tries to jailbreak the agent, inject instructions, or leak another account’s context, not a generic content filter. The team’s point is that safety is not a stage. It runs through the lifecycle.
Keep a second distinction in view. A harmfulness score from an evaluator is a judgment about what already happened. A guardrail is a synchronous block. Scoring a bad answer after the fact is not the same control as refusing it on the way out.
What three months of scorecards actually moved
Keep the numbers apart. They share endpoints. They are not one result.
Over three months of targeted iteration, the team reports the scorecard pass rate rising from 93 percent to a stable 96 percent. That rate is on a curated set, not on live traffic. Four percent of scorecard cases still fail. The write-up does not break those remaining failures into types, so they should not be briefed as “the leftover hallucinations.”
In a separate optimization example, replacing memorized few-shot values with placeholders is linked to two other moves: pure hallucinations falling from 138 to under 52 per 10, 000 responses, and grounding rate improving from 93 percent to 96 percent. Same change. Two measures. Two denominators: a rate per 10, 000 responses, and a percentage of figures traced to source material. The 93-to-96 pass-rate move is a third figure. Collapse any of these and you will brief a leadership team on a metric you do not have.
The authors say the gains confirmed that retrieval was sound and that the fixes were in generation. Take that as their diagnosis, not as an outside audit. The published figures do not include evaluator false-positive or false-negative rates, and they do not say how disagreements between judges were resolved. Judge models vary. The checks that do not flake are the programmatic ones. AgentCore includes trajectory matchers that compare the actual tool sequence with an expected one, use no judge model, and, as a practitioner guide current as of June 2026 notes, consume no judge tokens. The sales write-up does not claim those matchers produced the 96 percent.
A few limits belong in the same breath as the wins. The account does not say how many users or queries the system handles, which models it runs, or what business impact the prevented errors had. There is no independent validation of the improvements. A 3-point gain on a scorecard is a tightening, not a transformation.
There is also a cost principle the team states without tying it to a specific model comparison.
A model that improves accuracy by 5 percent but doubles latency and triples cost doesn’t automatically win.
Their evaluation window watches quality, latency, and cost together. Accuracy that blows the budget is a more expensive way to be a little more accurate. It is not automatically the release you ship.
What the platform is, and is not
Amazon Bedrock AgentCore Evaluations is a managed scoring service tied to the AgentCore runtime. It can score live interactions or past sessions, attach reasoning to each score, and surface results in CloudWatch’s generative-AI observability view. The runtime emits OpenTelemetry traces, metrics, and logs. Execution comes in three modes: continuous online scoring, a synchronous on-demand evaluation API, and asynchronous batch evaluation. A practitioner guide current as of June 2026 still had batch in public preview. Confirm the current status before you build a release gate on it.
The sentinel, the 200-question gate, the canary, and the 10 percent live sample are an internal operating model mapped onto those modes. They are not a product feature with a stage count. The team’s cadence is the useful part: define quality, compare changes, block regressions, watch live drift, enforce safety on every turn, and feed failures back into the golden set.
Built-in judges include helpfulness, correctness, faithfulness, instruction following, response relevance, tool-selection accuracy, tool-parameter accuracy, goal success, and harmfulness, among others. Launch announcements cited 13 built-in evaluators. As of June 2026, the developer guide listed 17: 14 LLM-as-a-judge evaluators and three trajectory matchers. The guide is the list that matters. A blog recap goes stale.
A practical starting set, if you are not already drowning in rubrics, is goal success, helpfulness, correctness, tool selection, tool parameters, and harmfulness. Add faithfulness and context relevance when the agent retrieves. Add trajectory matchers when you want a CI gate that does not wobble because the judge had a mood.
The same operating shape can fit other multi-agent systems. The failure taxonomy should not. A support agent’s “wrong entity” is not a seller’s “wrong entity.”
Questions worth asking before an agent briefs a forecast
-
Can ordinary monitoring tell you an AI agent is right?
No. In pre-launch evaluation, monitoring stayed green while the Customer Insights Agent inflated $410.08 into $410.08 million, relabeled a revenue metric, and presented calculations as sourced facts. The evaluation caught it before a seller could brief it. Uptime would not have.
-
Which mistakes should fail a sales answer outright?
The team fails a response for any one of six: metric swap, wrong period, wrong entity, pure hallucination, wrong symbols, or unauthorized calculation. Broader scores such as helpfulness still matter, but they do not excuse a wrong dollar.
-
Is a fast pre-merge check enough to ship?
Not in this design. A 15-case sentinel runs on every staging deployment, then a release gate runs 200 held-out questions. Fifteen cases cannot cover tens of thousands of query patterns, and the team says clearing the sentinel is necessary but not sufficient.
-
How fast can quality move when nobody shipped code?
Fast. The team reports customer-identification quality falling from 98 percent at 1:30 a.m. to 81 percent at 2:00 a.m. and 80 percent at 2:30 a.m., driven by tickers, shorthand, and typos. Weekly data refreshes can move quality with no release at all.
-
Did the scorecard produce a dramatic accuracy jump?
The team reports scorecard pass rate moving from 93 percent to a stable 96 percent over three months, and, in a separate generation fix, pure hallucinations falling from 138 to under 52 per 10, 000 responses. That is a tightening on the measures they published. A grounding-rate move from 93 percent to 96 percent is a different metric, and none of these figures is an audited count of decisions saved.
-
What should a leader do when 4 percent of held-out cases still fail?
The write-up does not say, and a 96 percent scorecard pass rate is not live accuracy. Read the failing cases before the next forecast meeting and decide whether that class of miss is acceptable to brief. The scorecard will not make the call.
-
Should the most accurate model automatically win?
Not on their principle, which is not a measured bake-off: a 5 percent accuracy gain that doubles latency and triples cost does not automatically win. Quality, speed, and cost belong in the same evaluation window.
-
Is AgentCore Evaluations itself the operating system?
No. It is the managed scoring layer (online, on-demand, and batch) with built-in judges, custom evaluators, and trajectory checks. The cadence, the 15-case sentinel, the paging rule, and the sales failure taxonomy are how this team operates the service.
Steal the operating rhythm. Define the quiet miss, keep a golden set that changes when the business does, refuse to ship on a 15-case smoke test, and page a human when live scores bend for an hour and a half. Write your own list of quiet lies. An agent becomes trustworthy when a four-hundred-dollar fact cannot be briefed as four hundred million, because the scorecard will not let it.