AI and COBOL: test the result, not just whether the rewrite runs

OpenAI MultiLangBench repository contributors, Apache License 2.0. Selected README excerpt; typesetting and arrows added by PING. Documentation excerpt; no COBOL implementation or benchmark result shown.

A rewritten program can run without preserving the job it was meant to do. OpenAI's 19 October research announcement reports progress on building software in less well-represented programming languages and migrating COBOL programs. The useful question for a team considering such tools is not simply whether the new code starts. It is whether the outputs and business rules that matter still match.

Two evaluations, two different questions

OpenAI separates MultiLangBench, which asks models to build SQLite-compatible database engines across twenty programming languages, from a COBOL migration evaluation that checks preservation of observable behaviour. It reports gains in both. These are the supplier's evaluations, not PING's tests, and neither establishes that a particular organisation's application can be safely replaced.

The published MultiLangBench repository makes one boundary inspectable. Its README describes an initial database, a query run by the candidate implementation, and a validation query checked with reference SQLite. It also distinguishes the converted test collection from a smaller recommended evaluation subset. A result needs its test selection and setup beside it, not just a headline score.

Write down what must not change

Before considering a trial, ask the application's responsible specialist to identify a small, permissioned set of behaviours worth preserving. Start with an ordinary record, an incomplete record and a boundary case that has previously caused confusion. For each, write the expected result and why it is correct. Do not put customer data or a live transaction system into an experiment merely to make the demonstration realistic.

Ask which differences are acceptable and which are failures. A changed label might be intentional; a changed amount or omitted record may not be. Keep those decisions with the examples so that a reviewer is not forced to infer the intended rule from whichever output happens to look plausible. This is a proposed review method, not evidence that any migration tool has passed it.

Look beyond a familiar-looking answer

The announcement specifically identifies rounding and missing-record handling as behaviours that can matter in migration. Separately, IBM's COBOL explainer describes the language's fixed-point decimal support. That makes the rule behind a number a better question than whether the displayed number looks sensible. Have an appropriately qualified maintainer define the expected handling rather than asking an assistant to invent the policy.

For a proposed demonstration, retain the same input, the old result, the candidate result and the check used to compare them. Record the actual model and settings, the code revision and the environment. If the old result is itself disputed, stop treating it as an unquestionable answer: resolve the business requirement first. A matching pair of outputs can preserve an existing mistake too.

The program is part of a larger system

IBM also cautions that modernisation involves the platform and integrations, not language translation alone. A narrow example does not cover every file, scheduled job or downstream consumer. Ask the responsible team to name what reads the output and what would happen if a field changed, a record was missing or a job ended late. Those are investigation questions, not a claim that PING has audited a system.

For teams in Lagos, Nairobi, Johannesburg or London, do not assume the public benchmark represents their institution's code, approvals or data-sharing permission. A vendor result is a reason to investigate a bounded problem, not authority to connect an assistant to a production database. Existing access and change controls should govern any separately approved trial.

Keep the result, its test and its limits together. The next practical step is a clear preservation checklist reviewed by the people responsible for the application, not an immediate rewrite. PING has not run either evaluation or performed a migration. Find more source-led technology explainers in PING's archive.