OpenAI GPT-6.1 Sol Boosts COBOL Migration Accuracy to 97.9%

OpenAI has released new models, GPT-6.1 Sol and GPT-6 Astra, significantly improving their ability to understand and generate low-resource languages such as COBOL. In migration tasks, accuracy has surged from 5.4% to 97.9%.

The primary challenges in migrating COBOL systems stem from a lack of semantic understanding and insufficient test verification; low-resource languages suffer from a scarcity of high-quality training data; and enterprise codebases are highly customized, limiting the generalization capabilities of general-purpose large models.

This article is suitable for enterprise architects, legacy system maintenance engineers, and AI engineers.

How many people today can still read code written decades ago?

In banks, insurance companies, manufacturing firms, and public service institutions, some core business systems have been running for decades. They may use programming languages like COBOL and Fortran that are no longer popular. The original developers have long since left, technical documentation is often incomplete, and many business rules exist only within the code itself.

These systems do not necessarily fail frequently, but they are often difficult to modify. For enterprises, rewriting a system can sometimes be far more challenging than developing a new one from scratch. Development teams must not only understand code from decades past but also ensure that after migration, accounting calculations, data processing, and business behaviors under various exceptional conditions remain consistent with the original system.

The emergence of AI coding tools once raised hopes that these issues could be resolved. However, over the past few years, large models have remained most proficient in mainstream languages like Python and JavaScript. When encountering historically significant programming languages with scarce public training data, or highly customized enterprise codebases, their performance has often been unreliable.

On October 19, OpenAI published new research on low-resource programming languages, attempting to change this situation.

In a COBOL code migration benchmark, the previous generation GPT-5.6 Sol achieved an accuracy of only 5.4% at low reasoning intensity. The latest GPT-6.1 Sol reached 86.6% at the same reasoning intensity, further climbing to 97.9% when reasoning intensity was increased. Meanwhile, OpenAI also released a test covering 20 programming languages, requiring AI to implement a SQLite-compatible database engine from scratch. GPT-6 Astra achieved an average test pass rate of 87.4%, compared to just 26.8% for the previous generation GPT-5.6 Sol.

These figures indicate that AI Coding capabilities are expanding into programming languages that were previously relatively weak areas. But even if AI can now complete complex cross-language code migrations in benchmarks, can enterprises truly feel confident handing over legacy systems that have run for decades to AI?

OpenAI Begins Challenging the Weak Links in AI Programming

The focus of this OpenAI research is on so-called "Low-Resource Programming Languages." Here, "low-resource" does not mean these languages are unimportant, but rather that compared to popular languages like Python and JavaScript, there is relatively limited public code, technical documentation, and other training materials available for models to learn from.

COBOL is a typical example. This programming language, born in 1959, still plays a crucial role in global financial transactions and mainframe systems. Citing IBM data, OpenAI noted in the article that COBOL supports a vast number of online banking, credit card transaction, and ATM systems. Additionally, Fortran is still used in weather forecasting, scientific computing, and nuclear reactor safety analysis; languages like Visual Basic and OCaml also have their own existing systems and application scenarios.

For AI models, these languages present two distinct types of challenges: one is whether they can write complex new programs using them, and the other is whether they can accurately understand existing programs and migrate them to another language.

To measure these two capabilities separately, OpenAI designed two benchmarks. The first is named MultiLangBench, which requires models to build a SQLite-compatible database engine from scratch in a specified programming language. This is not simply generating a few SQL query snippets. Models need to continuously organize the codebase, manage state, handle exceptions, add features, and ensure that previously completed functions do not break during subsequent modifications.

The benchmark uses database files as a unified interface: the program under test receives an initial database and SQL queries, generates a modified database, and then compares it against the SQLite reference implementation. This allows programs generated in different languages to undergo the same set of functional tests.

According to results published by OpenAI, GPT-6 Astra achieved an average test pass rate of 87.4% across 20 programming languages. Specifically, OCaml improved from 25.6% in the previous generation to 85.8%, Fortran rose from 21.4% to 83.5%, and Visual Basic jumped from 27.5% to 89.4%.

However, COBOL remains the most difficult language among them. GPT-6 Astra’s pass rate on COBOL was 72.1%. Although this is significantly higher than the previous generation's 10.9%, a gap still exists compared to other languages.

Notably, OpenAI also made the MultiLangBench evaluation code and test suite publicly available, allowing developers to review the testing methods and results for each language.

Compared to common algorithm problems and single-file programming tests, this benchmark is closer to the continuous implementation, debugging, and regression testing found in real-world software development processes. However, it still operates in a controlled environment and cannot directly equate to the development capability required for large-scale enterprise systems.

The second benchmark is the most noteworthy part of this research.

COBOL Migration Accuracy Reaches 97.9%—Where Exactly Lie the Difficulties?

OpenAI’s second benchmark requires models to migrate existing COBOL programs to C++, Java, Kotlin, Python, or Rust while preserving the behavior of the original programs.

On the surface, this appears to be a code translation task. But for developers who have actually performed legacy system migrations, successful compilation is still a long way from successful migration.

Take financial systems' amount calculations as a common example. COBOL has specific semantics for fixed-point arithmetic. In certain cases, if the program does not explicitly specify rounding, excess precision is directly truncated. Suppose the original program calculates amounts according to this rule, while the migrated Java or Python program uses a different default rounding method. The new program might have no syntax errors and execute normally, yet produce different calculation results.

For ordinary applications, such discrepancies might be just an inconspicuous bug; however, in scenarios like financial settlement and account reconciliation, subtle numerical deviations can lead to severe consequences. Similar issues include how missing records are handled, how abnormal inputs are responded to, how strings and numbers are converted, and the program's dependencies on existing data formats.

Often, these behaviors are not documented in design specifications but are hidden within long-running code. More troublesome still, some implementations that seem unreasonable may precisely embody historical business rules that enterprises must preserve.

Therefore, OpenAI did not simply use "successful code generation" as its evaluation standard. Instead, it used hidden tests to check whether the observable outputs of the programs before and after migration were byte-for-byte identical. Only by passing all tests is a migration task considered successful.

According to the results published by OpenAI:

• GPT-5.6 Sol achieved a migration accuracy of 5.4% at low reasoning intensity.
• GPT-6.1 Sol improved accuracy to 86.6% at the same reasoning intensity, with an estimated 39% reduction in API costs.
• GPT-6.1 Sol further reached 97.9% accuracy at higher reasoning intensity.

This means that, at least in the evaluation tasks constructed by OpenAI, the model can not only convert old code into a new language but also maintain the observable behavior of the original program in a high proportion of tasks.

However, 97.9% accuracy does not mean that "97.9% of the code in large COBOL systems can be correctly migrated," nor does it imply that banks can hand over their core business systems to AI for automatic rewriting. This figure measures the success rate of specific tasks under hidden tests. The scale, complexity, external dependencies, data states, and runtime environments of real enterprise systems may far exceed the coverage of these evaluations. Even if the migrated program passes existing tests, it cannot prove that it is completely consistent with the original system under all unknown inputs and business boundary conditions.

OpenAI explicitly stated in its research that applying these capabilities to production software still requires expert review and testing against business rules and system dependencies.

Nevertheless, the significant improvement in accuracy from 5.4% to 97.9% remains a progress that cannot be ignored. In the past, models might have struggled to even complete a relatively comprehensive migration task correctly; now, the question is gradually shifting to: What kind of verification system is needed for model-generated results to enter the production environment?

A Year Ago, AI Could Only Help Programmers Understand Legacy Code

In fact, using AI to modernize legacy systems is not a newly emerged direction. In June 2025, InfoQ reported on the practice of Morgan Stanley's internal AI tool, DevGen.AI. At that time, Morgan Stanley disclosed that this tool had helped development teams review approximately 9 million lines of legacy code, saving about 280,000 hours of development effort cumulatively.

Further Reading: “Tackling 9 Million Lines of ‘Spaghetti Code’ and Saving 280,000 Hours! AI Programming Takes Aim at ‘Ancient’ Programming Languages”

DevGen.AI is based on OpenAI’s GPT models and was built by Morgan Stanley’s internal development team. Unlike typical AI coding tools, one of its most important capabilities is converting programs written in legacy languages like COBOL into natural language business specifications. This helps developers understand system functionality, allowing engineers to refactor based on these specifications.

Behind this approach lies a very practical reason: As fewer developers become familiar with certain old languages, even if enterprises find engineers proficient in Java or Python, they may not be able to directly understand COBOL code from decades ago.

By leveraging AI to generate business specifications, originally obscure and difficult-to-understand programs can be transformed into descriptions that are easier to review and discuss. Developers can thus sort out system functions, locate business logic, and formulate next-step migration plans.

However, Morgan Stanley acknowledged at the time that AI still had obvious shortcomings in directly migrating complete code. Even if it could reimplement the original program in a new language, it might not fully utilize the features of the target language to generate efficient and maintainable code.

This perfectly illustrates two distinct levels of problems faced in legacy system modernization. The first problem is understanding: Enterprises need to clarify what the existing system actually does and which rules within it still need to be preserved. The second problem is migration: After achieving understanding, development teams still need to reimplement these functions using modern technologies and verify whether the behaviors of the new and old systems are consistent.

Morgan Stanley’s practice proves that AI can already help engineers overcome the first hurdle. Meanwhile, the evaluation published by OpenAI demonstrates progress in large model capabilities regarding the second problem.

Of course, the two cannot be simply compared quantitatively. DevGen.AI is an actual engineering tool running within an enterprise, targeting real codebases with complex dependencies; OpenAI’s new research showcases model capabilities on standardized evaluation tasks. But placing these two practices side by side, we can clearly see that the technical path for AI involvement in legacy system transformation is broadening: From code explanation and business knowledge extraction to code generation, migration, and behavioral consistency testing, AI can take on more and more stages.

For enterprises, the next consideration is how to organize these capabilities into reliable, auditable engineering processes.

Without COBOL, Enterprises Still Have Old Systems That Are Hard to Change

COBOL is a typical case of legacy system modernization, but the similar challenges enterprises face go far beyond this single programming language.

A Java monolithic application that has been running for over a decade may have accumulated millions of lines of code, involving numerous interfaces, database tables, and complex external dependencies. Engineers who initially participated in development have left one after another, technical documentation has not been updated for a long time, and some business logic has undergone multiple rounds of modifications, making it difficult to explain why it was implemented in a certain way.

These systems may not necessarily use outdated technologies and might still stably support the enterprise’s core business today. However, every time architectural upgrades, database replacements, or service splits are performed, development teams face the same question: Will modifying a piece of code break a business rule hidden within the historical implementation?

Some modernization efforts are even more complex than cross-language migration. For example, when decomposing a monolithic architecture into microservices, business operations that originally relied on local transactions may require redesigning distributed transactions, data consistency mechanisms, and exception compensation strategies. Even if every service in the new system passes unit tests, this does not directly prove that the overall business process remains consistent with the legacy system.

This is why the value of OpenAI’s recent research extends beyond improving accuracy in COBOL migration. The demonstrated capabilities in cross-language code understanding and generation, along with the approach of validating program behavior through executable tests, could be applied to broader enterprise system modernization scenarios. Examples include extracting business rules from legacy code, analyzing module dependencies, generating regression tests, or assisting in rewriting specific modules while comparing behavioral differences between old and new implementations.

Of course, the benchmarks published by OpenAI are not yet sufficient to prove that models can independently handle large-scale enterprise system modernization. Issues such as cross-service dependencies, complex database states, concurrency, and fault handling are far more intricate than input/output testing for individual programs.

However, this research at least provides a noteworthy signal: As AI becomes increasingly proficient at understanding and migrating code characterized by scarce training data and complex semantic rules, its potential role in software modernization may extend beyond specific programming languages to encompass wider enterprise system transformation scenarios.

AI Can Modify Decades-Old Code, But Who Dares to Deploy It?

As AI begins to tackle legacy code that was previously difficult to handle, a more practical question arises: Who verifies that the AI-modified programs are safe for production deployment?

Both evaluations released by OpenAI use executable tests as a key metric for assessing code generation quality. In COBOL migration tasks, models must not only generate code in another language but also ensure that the output of the new program matches the original byte-for-byte in hidden tests.

However, real-world enterprise systems are far more complex than benchmark environments. For business systems that have been running for over a decade or even several decades, the challenge often lies not just in whether the code can be successfully migrated, but in whether there exists a sufficiently comprehensive test suite capable of proving that the migration has not disrupted existing business logic.

Many legacy systems are difficult to modernize precisely because original design documentation is missing, test coverage is inadequate, and business rules are tightly coupled with code implementation.

Consider a system with tens of millions of lines of code, incorporating business rules accumulated across different regions and eras. Engineers find it nearly impossible to enumerate all possible inputs and runtime states in advance. Even if AI successfully completes the code migration, teams still need to answer critical questions: Which behaviors have been verified? Which edge cases remain uncovered? If the new and old systems produce different results, is it due to a migration error, or did the original system contain historical defects?

Behavioral consistency at the code level is only one aspect of modernization. Enterprises typically also need to reconsider data models, interface compatibility, transaction processing, performance, deployment environments, and security controls. Even if the new program produces identical business outcomes, its performance under concurrent loads, fault recovery mechanisms, and interactions with peripheral systems may differ from the original.

Therefore, in real-world projects, a more prudent approach remains phased implementation: First, leverage AI to assist in understanding the original code and dependencies, establishing baselines for business rules and tests; next, select modules with relatively clear boundaries for migration and run regression tests in isolated environments; finally, control modernization risks through parallel comparison of old and new systems, manual review, and gradual cutover.

In this process, AI does not necessarily need to rewrite the entire system in one go. It can take on separate tasks such as code analysis, test generation, migration implementation, and issue resolution, while critical business validation and deployment decisions remain the responsibility of engineering teams.

For enterprises currently advancing system upgrades, technology stack migrations, and architectural transformations, these capabilities hold significant practical value. They do not need to possess a COBOL system to face challenges such as incomprehensible legacy code, unverifiable business rules, and uncontrollable modernization risks.

From 5.4% to 97.9%, OpenAI has demonstrated significant progress in AI’s ability to perform specific code migration tasks. As models improve their understanding of more languages and complex codebases, enterprises may reduce certain costs associated with analyzing and modernizing legacy systems. However, translating high accuracy rates from benchmarks into reliable migrations in production environments still requires a complete engineering verification framework.

In the past, a major obstacle to legacy system modernization was the scarcity of personnel capable of reading and modifying old code. As AI coding capabilities improve, another long-standing issue will become more prominent: How can enterprises prove that the transformed system continues to correctly execute business rules accumulated over decades? This may be the next major challenge software engineering needs to solve as AI truly enters core enterprise systems.

References:

• https://www.infoq.cn/news/fduhDwYGaDod8EuEUyEW