In-depth analysis · About 5 min read
AWS checks agent quality across nine lifecycle stages, not just an uptime dashboard
An internal sales-agent case connects domain-specific evaluation, release replay and live monitoring; its thresholds remain workload-specific.
- Source published
- Published here
- Source checked
AWS’s October 16 engineering account describes a nine-stage evaluation loop for an internal multi-agent sales intelligence system. The interesting failure is an apparently healthy service giving a confidently wrong business answer. The proposed response combines testing at tool, individual-agent and orchestration levels with release gates and production checks. This is an operating method illustrated through AWS’s own system, not proof that enabling a service makes every agent correct or safe.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- A pre-production mismatch between coordinated changes reportedly reduced pass rate from 78% to 38%, despite normal infrastructure monitoring.
- The narrow CI sentinel contains 15 cases; the broader release gate includes 200 held-out end-to-end questions and frozen tool responses.
- Thirty-minute evaluation windows, 10% live-quality sampling and synchronous guardrails on 100% of traffic are different controls.
Business correctness needs its own failure taxonomy
The source describes an agent turning $410.08 into a claim of $410.08 million, mixing revenue definitions and inventing a recurring-revenue estimate. A successful request and low latency would not identify those errors. AWS organizes evaluation around safety, quality, performance, cost and operational control, at three levels: tools, agents and their combined workflow. Its quality taxonomy separates the wrong metric, period, customer, unsupported number, symbol and unauthorized calculation. Our reading is that correctness must include the user’s scope. A legitimate parent-company number can still be wrong for a subsidiary question. A generic helpfulness score therefore cannot replace business-specific reference data, permission checks or rules about which calculations are allowed.
Build evaluators and reference cases before comparing versions
The first stages define acceptable outcomes, thresholds and a versioned golden set before comparing candidate prompts, models or tools. AWS combines built-in evaluators for broad qualities with custom evaluators, third-party libraries and deterministic checks where the answer is known. LLM judges address dimensions such as relevance, but are not the sole source of factual truth. The golden set draws on production examples, past failures and constructed edge cases, and exists at all three system levels. Each candidate is compared on quality, latency and cost together. Our interpretation is that an improved headline score can conceal a worse subgroup or a more expensive route; the evaluation contract must say what trade-offs are acceptable rather than letting the model choose them.
A fast merge check is not the full release decision
The CI sentinel is deliberately small enough to run on every staging deployment, with 15 high-risk cases and thresholds across the evaluation dimensions. Passing it is necessary but insufficient. Before production, the broader gate tests a full candidate with 200 held-out questions and rejects regressions, failure to beat the production baseline or too many individual low-scoring answers. Tools are first validated on known inputs; their correct responses are then frozen for replay. That isolates agent reasoning from changing live tools and makes candidate comparisons more controlled. It does not certify live dependency behavior. Scope mismatches can fail even when each underlying tool returns a valid number, so orchestration needs tests beyond checking components separately.
Live checks use different schedules and coverage
The article describes batch checks every 30 minutes, with consecutive windows used to filter noisy alarms and a daily full golden-set run. Three breached windows can page on-call staff; a single weak window is an operational signal rather than automatically an incident. Separately, online quality evaluation samples 10% of live traffic to find inputs the reference set missed. Synchronous input and output guardrails cover 100% of traffic in this implementation. These percentages cannot be interchanged: sampled quality judging is not universal fact verification, and content filtering is not a substitute for correctness evaluation. Upstream schema or retrieval changes can degrade answers without a deployment, which is why the release gate alone is insufficient.
Feedback must preserve evidence and remain governed
Execution tracing and evaluator scores have separate pipelines. The account requires CloudWatch Transaction Search setup for its tracing path, stores evaluation records and uses role-appropriate dashboards. Low-scoring traces can become permanent regression cases or prompt and tool-description recommendations. The source marks multiple dashboard figures as illustrative, so we do not convert those graphics into independent performance benchmarks. Our analysis is to retain versions of data, tool replies, prompts and evaluator definitions together, review changes before promotion, and budget the evaluation itself. A recommendation optimized for one judge still needs regression checks elsewhere. The article also identifies dataset evaluation as public preview; that status should not be generalized to every AgentCore capability.
Why it matters
For operators, the useful distinction is between service availability and answer validity. This case makes release decisions and incident diagnosis more concrete by linking an error to the tool, data scope or synthesis that caused it. Its evaluation loop complements external authorization controls; it does not replace them.
Limits and availability
We read the complete official article, without deploying agents, accessing internal sales data or reproducing its results. Thresholds and examples belong to AWS’s workload. Illustrative charts are not independent measurements. Evaluation, storage and monitoring consume resources; the article does not establish a free plan or guaranteed elimination of wrong answers.
Sources and evidence
Source published: Oct 16, 2026 · Source checked: Oct 20, 2026
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.