In-depth analysis · About 5 min read
Qwen on Qoder CLI: five workloads show why one score cannot rank every agent
Alibaba’s October 15 comparison reports useful engineering trade-offs, but each model-task generation was a single run on one harness.
- Published here
- Source published
- Source checked
Alibaba Cloud’s October 15 report compares Qwen3.8-Max, Claude Opus 4.8, GPT-5.6 Sol and Claude Fable 5 on five demanding workloads through Qoder CLI. It describes autonomous execution, generated artifacts and evaluation scripts. It also explicitly limits the conclusion: these are model-plus-harness results, with one run per task, not a general model ranking.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- The workloads cover a procedural 3D scroll, a Rust key-value server, four frontend prompts, frozen OpenAlex analysis and a 92-second narrative animation. External models used an experimental setup, not their shipping products.
- All models reportedly passed 27 private KV-store correctness tests. Small throughput differences among the top three were below reported cross-run variation; repeating performance grid points five times does not repeat agent generation.
One harness controls the comparison, but also limits it
The report gives the four exact models the same Qoder CLI tools, execution environment and feedback loop. Each receives a single prompt and runs without follow-up prompting; some tasks last hours. This reduces one source of variation but measures a particular model-and-tool combination. The external models run through an experimental arrangement, so results should not be read as tests of their usual consumer or developer applications. The author says prompts, data, deliverables and scripts are public, but we did not inspect those linked artifacts. Crucially, the report states one generation per task: it does not estimate the distribution of outcomes from repeatedly sampling an agent.
The 3D scroll exposes a mismatch between metrics and appearance
The scroll task asks for six chapters and 24 camera positions, built from procedural solid geometry without placing the original painting behind the scene. All four reportedly produced real 3D. Qwen scored 63/100 against Fable’s 73 and took 263 minutes, with roughly 378 screenshots used for iteration. The author describes Qwen’s custom renderer as efficient but its scene as sparse. Repeatedly tuning grain increased an image-similarity metric while making the picture worse to a human viewer. AISubHub analysis: an optimization target can reward the wrong visual feature. That is a failure mode in this example, not proof that every image metric or every Qwen output behaves this way.
Rust correctness, throughput and tail latency are different outcomes
All four implementations reportedly passed 27 private correctness tests. Qwen’s peak read, write and mixed rates were 11.8M, 12.2M and 10.2M operations per second. The report links much of the gap to batching responses rather than writing once per command. But it explicitly says the top three’s 0.5–9% ordering differences were smaller than cross-run variation of up to ±55% and not statistically significant. Each performance grid point was measured five times and summarized by its median, with a four-core/3GB server and separate eight-core load generator. That repeats measurements of delivered code, not the agent’s generation. Low-load latency and overload stability also had different winners.
Four frontend prompts test presentation choices, not one universal style
The frontend workload contains weather cards, a coffee landing page, a procedural planet and music-synchronized visuals. The report distinguishes hierarchy, controls, illustration and completeness. For the weather card, Opus alone had zero critical axe accessibility issues yet omitted a temperature; a clean accessibility check did not ensure all requested content. Other examples contrast a rich scene with usable controls or restrained layout. We read these descriptions, without independently viewing every artifact or video. AISubHub analysis: acceptance should keep content, accessibility and interaction separate. A visually attractive screenshot can hide a missing value or a control that does not work.
OpenAlex rewards traceability without turning a sample into the world
The frozen dataset covers 2020–2025: 60 country-year units across ten countries, 1,198 sampled papers, and 290 PDF pages from 17 CC BY papers. The report says Qwen scored 100, Fable 94, Opus 89 and GPT 87 for the task, with Qwen passing all 44 automated checks. It particularly values page-level links and separating aggregate country-year statistics from the paper sample. Qwen reportedly avoided treating sample topic frequencies as global shares or correlations as causes. AISubHub analysis: these are useful acceptance dimensions, but the vendor’s index is not a general scientific-intelligence score. A dashboard’s evidence chain must still be checked against the underlying tables and pages.
Offline delivery can matter more than a one-point lead
The final workload is a 92-second procedural narrative animation. Fable scored 90 and Qwen 89; the report credits Fable’s visual polish but notes a network-loaded 3D engine, whereas Qwen bundled its engine for offline playback. This illustrates how delivery requirements differ from aesthetic scoring. A model may reach the end of execution while leaving an artifact requirement unmet; the scroll example even includes an unfinished report from one competitor. AISubHub analysis: reproduce relevant tasks with the exact versions, environment and acceptance rules before selecting a tool. These single-run case studies explain trade-offs, not the current global order of models.
Why it matters
AISubHub analysis: a useful comparison separates correctness, latency, throughput, visual quality, traceable evidence and offline delivery. Finishing an agent run does not prove every acceptance requirement was met.
Limits and availability
We read the complete vendor report, not the linked repository, raw artifacts or videos, and did not reproduce its tests. Scores, timings and visual judgments remain attributed to the report. Its specific model versions and experimental harness cannot establish the standing of every current model or product.
Sources and evidence
Source published: · Source checked:
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.