In-depth analysis · About 4 min read
OpenAI reports progress on coding in less common languages
New evaluations examine SQLite implementations and behavior-preserving COBOL migration, with results reported by OpenAI.
- Source published
- Published here
- Source checked
OpenAI published new coding evaluations on October 19, focusing on languages with less available training material, including COBOL, Fortran and Visual Basic. The announcement concerns measured engineering tasks, rather than a guarantee that legacy systems can be migrated automatically.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- MultiLangBench asks models to build SQLite-compatible database engines in 20 languages. OpenAI reports an average test pass rate of 87.4% for GPT-6 Astra at maximum reasoning effort, versus 26.8% for GPT-5.6 Sol.
- A separate COBOL migration evaluation checks whether rewritten programs preserve observable behavior. A migration counts as successful only when every hidden test passes.
What the two evaluations actually measure
The October 19 release separates two engineering problems that are often bundled together in AI coding claims. MultiLangBench measures how much new database functionality a model can implement; the COBOL task measures whether a rewrite preserves an existing program’s behavior. In our reading, that distinction matters more than a single headline percentage. A system can generate plausible source code and still fail either test. For a team modernizing older software, the relevant question is which skill its project needs, and what evidence would reveal a failure before deployment.
How the SQLite test checks working software
In MultiLangBench, the candidate receives a database and a SQL request, modifies the database, and is checked using the real SQLite implementation. The released README specifies SQLite 3.40.1 and three components: the initial database, the candidate’s query, and a reference validation query. It reports converting about 24,000 of SQLite’s 51,000 TCL tests, with a default evaluation subset of 272. Running the larger suite must be disclosed. Our interpretation: this checks observable functionality across languages, but the headline result should not be read as coverage of every original SQLite test.
Reading the scores alongside their settings
OpenAI reports an average 87.4% test pass rate for GPT-6 Astra at maximum reasoning effort, versus 26.8% for GPT-5.6 Sol. COBOL improves from 10.9% to 72.1%, remaining harder than the other tested languages. GPT-6 Luna averages 40.3%, with estimated API spending of $0.77 per sample versus $3.48 for GPT-5.6 Sol. These are different accuracy-cost tradeoffs, not interchangeable products. Our analysis is to compare cost per accepted result, including failed attempts and review time, rather than assume a cheaper sample always makes a migration cheaper.
Why a successful rewrite must preserve behavior
The separate migration evaluation translates COBOL into C++, Java, Kotlin, Python, or Rust. Hidden tests require byte-for-byte agreement in observable outputs, and all tests must pass for success. The announcement gives a concrete trap: COBOL fixed-point arithmetic discards excess decimal digits unless rounding is requested. A rewrite that rounds instead may compile and run while changing financial results. OpenAI reports GPT-6.1 Sol at 86.6% accuracy with low reasoning effort, versus 5.4% for GPT-5.6 Sol at the same effort and 39% less API cost; higher effort reaches 97.9%.
What a meaningful reproduction would require
The public repository asks evaluators to remove existing SQLite implementations from the model environment and disallows internet access. It treats wrong-language or mixed-language solutions as failures; mismatches, crashes, and tests exceeding 60 seconds also fail. These conditions guard against confusing implementation with reuse of an installed engine. We inspected the documentation, not an executed benchmark. A local reproduction should record the exact model, reasoning setting, test subset, environment, and cost, so that a changed setup is not silently presented as the same experiment.
A practical path from benchmark to migration
Our suggested next step is a bounded pilot around one noncritical module, with representative inputs and expected outputs captured before asking the model to rewrite anything. Add cases that stress rounding, missing records, and the dependencies relevant to that module; those examples are evaluation advice, not results from this release. Review differences with domain owners and keep rollback possible. The announcement points developers to API experiments, but does not grant a ChatGPT subscription any new API budget. Strong benchmark progress supports investigation; it does not replace production acceptance criteria or expert responsibility.
Why it matters
For teams maintaining older software, the useful next step is to inspect the released evaluation setup and test representative business rules. Choosing a model should account for test coverage, reasoning settings and actual API costs.
Limits and availability
These are vendor-reported results, not an AISubHub replication. Benchmark success does not establish production safety; expert review and testing against business rules and dependencies remain necessary.
Sources and evidence
Source published: Oct 19, 2026 · Source checked: Oct 20, 2026
Further reading
Source checked: Oct 20, 2026
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.