In-depth analysis · About 6 min read
Decision-model evals: why a 0.50 threshold can hide failures
Hamel Husain and Antaripa Saha compare six classifiers, separating threshold tuning, validation, API cost and probability calibration.
- Published here
- Source published
- Source checked
Hamel Husain and Antaripa Saha’s October 20 original evaluation asks how to turn decision-model scores into useful quality checks. Its lesson is a workflow rather than a universal winner: define the failure, set an acceptable precision–recall tradeoff, choose thresholds without using the held-out labels, and check probabilities separately from binary decisions.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- The authors start with 100 apartment-leasing agent traces, exclude two internal QA examples from performance scoring, and evaluate 98 traces containing 61 labeled failures and 37 without a labeled failure.
- Reported tuned F1 is 0.79 for Perplexity at 0.82 US cents per 100 eval runs and 0.87 for OpenAI Decisions at 3.72 cents. Costs cover 100 runs; performance uses 98 traces, and neither is a general model ranking.
Define what is being classified before reading a score
The study evaluates whether a complete apartment-leasing assistant trace contains a substantive error, such as a failed booking or missed human handoff. It tests Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions using GPT-6 Luna. Models receive the same criteria, with equivalent structured criteria for Jev. The reference has 61 known failures: 39 from the original human review plus 22 subsequently found by Braintrust and Arize and manually validated. The other 37 have no labeled failure. That wording matters: the experiment compares with this reference, not an omniscient declaration that every possible issue has been found.
A threshold is an action rule, not a model property
Each model assigns a score between zero and one, and a threshold turns it into an alarm. The authors illustrate a classifier scoring failures at 0.40 and passes at 0.10: a 0.30 threshold separates them, whereas the default 0.50 catches no failure. Lowering a threshold generally trades more detected failures for more false alarms. Precision asks what fraction of alarms are known failures; recall asks what fraction of known failures are caught. F1 is their harmonic mean and treats the two symmetrically. Our analysis is that a support escalation or destructive-operation gate may have different error costs, so choosing F1 is itself a decision, not a neutral default.
Held-out validation does not erase infeasible training splits
The article uses stratified five-fold cross-validation, selecting thresholds on training examples and evaluating other examples with shared splits across models. For candidate models it seeks the best F1 subject to at least 80% precision and 80% recall on training data. A crucial footnote says that when no threshold qualifies, the reported experiment falls back to an unconstrained F1 optimum and retains validation predictions. That happened in 36 of 100 training splits for Perplexity and 29 for d1-ultra, and never for OpenAI. Thus the displayed averages do not certify the 80% requirements. The article’s suggested operational prompt instead asks the system to report infeasibility rather than relax a requirement; that stricter instruction should not be confused with how its illustrative results were calculated.
Keep the cost denominator separate from performance
Among the remaining candidates, the authors report Perplexity at F1 0.79 for 0.82 US cents per 100 eval runs, versus OpenAI Decisions at 0.87 for 3.72 cents, about 4.5 times the cost. The cost calculation covers all 100 API runs; performance excludes two internal QA traces. Costs come from saved token use and provider rates, including repricing Liquid using October 17 rates, rather than a universal future bill. Threshold tuning reuses saved scores without new model calls. The main comparison deliberately sets latency aside; separately reported mean response times came from different runs and do not establish comparative throughput or tail latency. A production decision therefore still needs its own batching, concurrency and network measurements.
A probability requires a different check
A score of 0.80 is a useful probability only if roughly 80% of similar-scored cases actually fail. Calibration evaluates that relationship, separately from choosing a pass/fail threshold. The article groups this small sample into five reliability groups of 19 or 20 traces and fits Platt scaling on training folds. One Clef Flash group averaged about 8% predicted risk while 60% had failures. Adjustment helped some models, but not all: OpenAI’s Brier score, where lower is better, changed from 0.159 to 0.164 with the illustrated log-odds fit. Another exploratory fit gave 0.154. Those outcomes show sensitivity to fitting choices, not a guarantee that calibration improves every model or that a future individual case has an exact known risk.
A narrow, versioned evaluator is the transferable result
The authors recommend separate judges for particular failure modes rather than retaining the broad test forever. Our proposed application is to version the criteria and human labels, keep an untouched final test set, choose business-specific precision and recall requirements, and explicitly surface splits where no threshold qualifies. Prompt changes can require fresh threshold and calibration checks. A cheaper model can then be assessed against a stronger baseline on the same task, rather than borrowed standings from this apartment-leasing sample. The article is a concrete original experiment, not a new availability announcement for all six APIs, an independently verified price catalog or evidence of the best general-purpose language model.
Why it matters
Our analysis: an inexpensive classifier is useful only after its errors fit the operational consequence. The portable contribution is the evaluation procedure and disclosure of infeasible threshold splits, not a claim that one provider always detects failures best.
Limits and availability
We read the complete original article and footnotes, not the private conversations or a new independent reproduction. The sample is small and task-specific, latency was not a controlled comparative result, and tuned thresholds or calibrated probabilities do not guarantee performance on future traffic.
Sources and evidence
Source published: · Source checked:
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.