In-depth analysis · About 6 min read
Perplexity’s agent-evaluation guide puts traces and calibrated judges behind the score
A six-step official guide explains how real failures, fixed test data and human-calibrated graders make agent changes assessable without treating one successful answer as reliability.
- Published here
- Source published
- Source checked
Perplexity’s October 22 guide addresses the engineering of agent evaluations, rather than announcing a model or reporting a new benchmark result. Its purchasing-agent example shows why teams need evidence about actions, repeatability and grading before interpreting a score.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- A plausible final answer can conceal unauthorized actions or leaked answers; success criteria must cover both the outcome and the path.
- Roughly 100 traces is an error-analysis starting reference, not a mandatory threshold. pass@k means at least one success in a stated k attempts.
- Repeat grading of the same saved run isolates judge variation; regression tests and production monitoring answer different questions.
What can a correct answer conceal?
An inventory assistant might propose a part that appears suitable while bypassing an approval rule. In Perplexity’s manufacturing example, an unavailable approved part or stale supplier information should trigger an appropriate response, not an unsupported substitute. The guide asks teams to define both successful work and unacceptable ways of achieving it. Domain experts must check that these criteria reflect the actual job. A fluent answer alone cannot establish that the agent respected its permissions.
The evaluation target therefore includes retrieved information, tool calls and intermediate decisions. Task success, action correctness and answer quality measure different properties; reliability concerns repeated trials and edge cases. Time, tokens and tool calls also matter when calculating the cost of successful work. Our analysis: a release decision needs an acceptance contract for the workflow, not merely a preference for a well-written response.
Which failures belong in the test set?
The guide starts with real execution traces: inputs, retrieved evidence, actions and outputs. A domain expert notes the first meaningful error in each run, then related observations are grouped into failure modes. An LLM may suggest categories, but frequency and severity determine priorities. Important failures become reusable cases with their input, starting conditions, expected outcome and grading criteria. This preserves the problem being repaired instead of substituting a convenient synthetic task.
Roughly 100 traces is offered as a practical reference, not a universal sample requirement or confidence guarantee. Reviewing can stop earlier at theoretical saturation, when further traces cease revealing meaningful new failure patterns. Before launch, synthetic examples can seed the collection; real use should inform it when available. Standard benchmarks can supplement this work, but unexpected passes or failures warrant investigation rather than unquestioning acceptance of their answer keys.
How can two agent versions face the same problem?
The illustrated folder separates a saved maintenance request from fixed inventory and approved-parts data. maintenance_request_01.json represents the input and expected outcome; inventory_snapshot.json and approved_parts.json provide comparison data. run_evals.py executes and checks the workflow, while results stores outputs, traces and pass/fail judgments. This is an illustrative test layout, not a published production system or software package that we executed. Our analysis: changing the inventory between comparisons would change the task as well as the agent.
Runs should retain retrieval steps and tool actions, not only final text. Because outputs can vary with identical inputs, cases need repeated trials selected according to risk, frequency and repair difficulty. pass@1 reports single-attempt success; pass@k concerns at least one success across a specified k attempts. State k explicitly. Finding one correct answer among several neither guarantees success every time nor proves that the agent can select the correct attempt.
Who should judge each requirement?
For exact conditions, the guide favors code-based checks: whether a part is approved, an output field is valid, tool arguments are correct or a final database state matches expectations. These checks offer a clear rule but cannot reliably settle every subjective decision. Context-dependent questions, such as whether an explanation adequately justifies a failed replacement, can use an LLM with an explicit rubric. A rubric must define acceptable behavior and the evidence supporting a verdict.
Domain experts remain necessary for specialist and disputed cases. Different checks within one run can use different graders; choosing an evaluation platform does not decide which failures matter to a business. Our analysis: separating permission checks from explanation quality makes a combined score easier to interpret. An LLM’s favorable reading of the prose must not silently replace an exact approval-list check. The guide supplies no measured accuracy gain for this division.
How can teams tell agent errors from judge errors?
The guide’s five-part calibration process collects inputs, outputs and traces; obtains independent domain-expert judgments; runs the LLM grader; compares decisions; and revises the rubric after investigating disagreements, before testing new examples. Human judgment serves as the reference in that process, not as a claim of infallibility. Calibration is therefore an evaluation of the evaluator, rather than another run of the agent being tested.
To test judge consistency, grade exactly the same saved run and files several times without rerunning the agent. Otherwise agent variation and grader variation remain mixed. A confusion matrix helps examine detected errors, missed errors and acceptable outputs incorrectly flagged. Manual review also discovers failure types that existing graders cannot recognize. The guide discusses benchmark problems through linked examples; we did not independently read those external studies or reproduce their findings.
When does a repair justify a release?
Reproduce the original failure from saved input, make a targeted prompt, model or workflow change, and confirm that the failing case is repaired. Then rerun the remaining suite to detect newly broken behavior. Baseline and updated versions must use the same cases and grading criteria. Assertions, test frameworks and CI/CD can automate these checks and apply release thresholds, but automation cannot establish whether the chosen criteria capture the product’s real risks.
After deployment, calibrated judges can assess sampled production traces, established failure categories can be tracked, and people can inspect unusual requests or new errors. Appropriate A/B tests add business evidence such as resolution or satisfaction, but do not replace evaluation or diagnose every underlying failure. Our analysis: the useful loop connects an observed failure to a repeatable check and a regression decision. This methodological guide establishes neither new product availability nor independently measured reliability.
Why it matters
Teams can use the guide to make agent revisions testable: preserve the failure, its evidence and its grader, then compare a repair against the broader workflow before monitoring real use.
Limits and availability
This is Perplexity’s methodological guidance and an illustrative procurement scenario. We reviewed the full article and its test-layout figure, not external studies, a deployed agent or platform entitlements; no accuracy, cost saving or universal pass threshold was independently measured.
Sources and evidence
Source published: · Source checked:
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.