← All posts
News

OpenAI's COBOL and Fortran tests: what GPT-6.1 Sol's 86.6% migration score does and doesn't tell legacy teams

OpenAI's new MultiLangBench and COBOL migration evals show big jumps for GPT-6.1 Sol and GPT-6 Astra. What the numbers cover, what they cost, and how to test on your code.

5 min read3 sources

OpenAI published two new coding evaluations on 19 October aimed at the languages models have historically been worst at: COBOL, Fortran, Visual Basic, OCaml and other “low-resource” languages with far less public code than Python or JavaScript. The headline number is a COBOL-to-modern-language migration test where GPT-6.1 Sol passes 86.6% of programs at low reasoning effort, against 5.4% for GPT-5.6 Sol at the same setting.

If your team maintains mainframe code, or sells legacy modernisation to banks and insurers, this is worth reading closely. It is also worth reading for what it does not show.

What OpenAI tested

MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in a given language. The program gets a starting database and a SQL query, and its output database is checked against reference SQLite. OpenAI says it adapted the tests from SQLite’s own test suite, and it covers 20 languages, from C, Go and Rust to COBOL, Fortran, Lean, Visual Basic and Zig.

The COBOL migration evaluation asks a model to port COBOL programs to C++, Java, Kotlin, Python or Rust. Hidden tests require byte-for-byte matching output, and a migration only counts if every test passes. OpenAI singles out COBOL’s fixed-point arithmetic, which truncates extra decimal digits unless rounding is asked for, as the sort of behaviour a careless rewrite silently breaks.

OpenAI’s framing for why this matters is that COBOL sits under more than 40% of online banking systems, and Fortran under weather forecasting and nuclear safety analysis. That figure is OpenAI’s own; it doesn’t give the underlying study.

The numbers

All figures below are from OpenAI’s post.

Test Model Score Compared with
COBOL migration, low effort GPT-6.1 Sol 86.6% GPT-5.6 Sol: 5.4%
COBOL migration, higher effort GPT-6.1 Sol 97.9% —
MultiLangBench, average of 20 languages (max effort) GPT-6 Astra 87.4% GPT-5.6 Sol: 26.8%
MultiLangBench, average (max effort) GPT-6 Luna 40.3% GPT-5.6 Sol: 26.8%

Per language, Astra’s biggest jumps over GPT-5.6 Sol on MultiLangBench were COBOL (10.9% → 72.1%), Fortran (21.4% → 83.5%), OCaml (25.6% → 85.8%) and Visual Basic (27.5% → 89.4%). COBOL is still Astra’s weakest language; OpenAI says it scores above 83% in the other 19.

Luna, the cheap tier, improves far less: COBOL goes from 10.9% to 17.1% and Fortran from 21.4% to 31.4%. OpenAI puts Luna’s cost at an estimated $0.77 per MultiLangBench sample against $3.48 for GPT-5.6 Sol.

On cost for the migration test, OpenAI says the low-effort GPT-6.1 Sol run costs about 61% of what GPT-5.6 Sol cost at the same setting.

What it costs to try

Model API ID Input / cached input / output (per 1M tokens) Access
GPT-6.1 Sol gpt-6.1-sol $2.00 / $0.10 / $10.00 API, Codex, ChatGPT Work plans
GPT-6 Astra gpt-6-astra $10 / separate cache rates / $50 API (initially a small set of organisations), Azure, AWS Bedrock; enterprise admins must switch it on

For migration work specifically, GPT-6.1 Sol is the model OpenAI reports results for, and it is a fifth of Astra’s per-token price. Astra’s numbers are on the from-scratch benchmark, which is a different task.

What this does and doesn’t tell you

It is a vendor benchmark. OpenAI built both evaluations and reports only its own models. There are no Claude, Gemini, DeepSeek or Qwen numbers to compare against. OpenAI says it is releasing MultiLangBench’s setup, tests and per-language results on GitHub, which means others can run it; until they do, treat the cross-vendor question as open. The post does not say the COBOL migration evaluation itself is being released.

Program-level parity is not system-level parity. A migration that passes hidden tests for one program says little about the parts of a mainframe estate that usually hurt: JCL and batch scheduling, CICS transactions, VSAM and DB2 data layouts, copybooks shared across hundreds of programs, and the undocumented behaviour that downstream reports depend on. OpenAI’s post doesn’t describe testing at that scale.

“Byte-for-byte” is the right bar. The useful lesson for teams is the test design, not the score. If you are evaluating any model for COBOL work, copy the approach: capture real inputs and outputs from production runs (masked), require exact matching on every observable output, and count a program as migrated only when all of its tests pass.

A practical way to evaluate this on your own code

  1. Pick 20–30 representative programs, including at least a few that do heavy decimal arithmetic (interest, GST, rounding of paise) and a few with complex file handling.
  2. Build golden outputs first. Run the existing COBOL on masked inputs and store every output file and report.
  3. Run GPT-6.1 Sol at low effort first, since that is where OpenAI reports its cost advantage, and move up to higher effort only for programs that fail.
  4. Diff mechanically, not by eye. Any mismatch is a failure.
  5. Run the same harness on at least one other vendor’s model. OpenAI’s numbers give you no baseline for anyone else.
  6. Track cost per passing program, not cost per token. A cheaper model that needs three retries is not cheaper.

Keep customer data out of prompts unless your contract and the provider’s data-retention settings allow it; OpenAI says Zero Data Retention is available to eligible API customers for Astra.

For teams in India

Legacy modernisation is a large line of business for Indian IT services firms and in-house bank technology teams. The practical takeaway is not that COBOL migration is solved, but that the cost of a credible first-pass translation has dropped enough to make a structured pilot cheap. The hard parts, such as test data, business sign-off on behaviour changes and the regulatory review that comes with core banking changes, are still where the time goes.

OpenAI is asking for feedback on which languages and workflows to prioritise next. If your estate includes PL/I, RPG or Natural, which aren’t in the 20-language list, that is worth telling them.

References

  1. Advancing Coding in Low-Resource Programming Languages — OpenAI, accessed 2026-10-20
  2. Introducing GPT-6.1 Sol — OpenAI, accessed 2026-10-20
  3. GPT-6 Astra — OpenAI, accessed 2026-10-20

This week in AI, every Sunday

One email with the releases that mattered and what they mean in practice. No spam.