COBOL modernisation with a local LLM: explain, document and convert to Java on-premise

Explaining, documenting and translating COBOL with a local LLM

A local LLM on a GPU workstation or server in your own data centre can explain COBOL programs, draft their documentation and business rules, and translate COBOL into Java, while the source code never leaves your network. Published research shows where it is reliable: in the COBOL-Coder study, the open model gpt-oss-120b translated COBOL tasks into Java that compiled in 98.6 per cent of cases, yet wrote compilable COBOL in fewer than one case in five. One NVIDIA RTX PRO 6000 Blackwell Workstation Edition or Max-Q with 96 GB runs that model in a workstation for a small modernisation team. Servers with four or eight RTX PRO 6000 Server Edition cards add team chat and night-time documentation runs over the whole mainframe code base.

This guide is for mainframe and core-system teams at banks, insurers, public bodies and retailers, whose source code and the business rules in it are data they often do not want to send to a cloud API. It covers the tasks in order of risk, the published COBOL benchmark results, IBM's on-premises route, the open-model route and the hardware for each stage. Our guide to GPU servers for banks and insurers covers the wider platform and DORA; our guide to local AI for PLC programming shows the same pattern for another legacy language. For SAP custom code on its way to S/4HANA, our guide to ABAP with a local LLM follows the same loop of compiler and unit-test feedback.

Four tasks, ordered by risk

1. Explain a program. The model reads the PROCEDURE DIVISION with the data definitions and copybooks it uses and describes what the program does, which files and tables it reads and writes, and which programs it calls. A wrong explanation misleads a reader but changes no code, so this is the first task to introduce.
2. Document the estate. The same pass, run over every program, produces descriptions, a data dictionary from the copybooks and summaries of business rules: the conditions in EVALUATE and IF blocks that decide interest, fees or eligibility. A COBOL specialist reviews each summary before it enters the documentation.
3. Write characterisation tests. The model drafts test cases and input data that record what a program does today. Run against the existing program, they become the reference for any later change or translation.
4. Translate to Java. The model drafts Java classes from a program and its data structures. The draft has to compile, pass the characterisation tests and go through your normal code review, exactly like code a developer wrote.

Writing or changing COBOL itself sits at the far end of the risk scale, and the research below explains why.

COBOL benchmarks for LLMs in published research

The most detailed public comparison is the COBOL-Coder paper by Concordia University and FPT Software AI Center, submitted to arXiv on 5 April 2026. The authors fine-tuned Qwen2.5-Coder at 7B and 14B on COBOL data checked with the GnuCOBOL compiler and tested code generation on COBOLEval, 146 tasks translated from the HumanEval benchmark, and translation on COBOL-JavaTrans, 143 COBOL and Java pairs of their own.

COBOL-Coder paper, arXiv 2604.03986v1, Tables 3 and 4, read on 30 October 2026. COBOL columns: writing COBOL on COBOLEval (146 tasks); Java columns: translating COBOL to Java on COBOL-JavaTrans (143 tasks). Pass@1 is the share of tasks whose first answer passes the tests. The values are the authors' own measurements.

For a project plan, the table reads like this. Reading COBOL and writing Java is a task the strongest open model in the study handles at least as well as GPT-4o. Writing COBOL is hard for general models: six of the open baselines, among them CodeLlama 7B and StarCoder2 15B, produced no compilable COBOL program on COBOLEval, Mainframer 7B compiled 69.17 per cent but passed the tests in only 6.16, and the domain-trained model reached about three compiling programs in four. Size alone decides little. The 14B Qwen2.5-Coder failed at translation, while the 16B DeepSeekCoder-V2 did well. The tasks are function-sized, far smaller than an enterprise program with dozens of copybooks, and the authors list as a limitation that each file is evaluated on its own, so shared copybooks and calls between programs are not captured. The COBOL-Coder weights are not published yet; the repository, under Apache 2.0, lists the 7B and 14B models as "Coming soon" and holds the benchmarks and evaluation code.

Researchers from IBM and IIIT Hyderabad presented a second approach at ICSE 2026: static program analysis infers Java classes, hierarchies and method signatures from COBOL data divisions and control-flow graphs, and the LLM translates the procedural logic within that design. On 20 GenAPP programs and 10 customer applications the pipeline reached a median structural score above 80 per cent and a functional score above 75 per cent, both rated by an LLM-as-judge framework; the abstract reports no compile or test rates. It describes the pipeline as "a core component of IBM's watsonx Code Assistant for Z (WCA for Z) product".

The open-model route on your own hardware

The open route combines a model server with tools that do the deterministic work. The model is the smaller part of the pipeline.

• Context assembly. A program's meaning depends on its copybooks, the JCL that runs it and the programs it calls. Resolving COPY statements and call chains is plain parsing, so a script assembles the context for each request; a search index over copybooks and JCL then answers questions across programs. Our guide to embedding and reranker GPUs for RAG covers that part.
• Model serving. gpt-oss-120b (117B parameters, 5.1B active, Apache 2.0) has 65.3 GB of weights and a context of 131,072 tokens. vLLM serves it with an OpenAI-compatible API on a workstation or a server; for a strictly isolated network, our guide to an air-gapped LLM server gives the offline installation.
• Checks. Every Java draft is compiled and run against the characterisation tests; every COBOL draft goes through your compiler. GnuCOBOL, the open compiler the COBOL-Coder authors used, catches syntax errors early, and your mainframe compiler remains the reference.
• A refine loop. OpenHands described, on 18 December 2025, a loop in which one agent converts COBOL files to Java and a second scores each conversion, repeating until the score passes a threshold. Compiler errors and failing tests are the most useful feedback for such a loop.

You can order the workstation alone, the setup of vLLM and gpt-oss-120b alone, or both together: the workstation delivered with everything installed and configured; write to us and we suggest the best solution for your tasks and budget.

IBM watsonx Code Assistant for Z on-premises

IBM's own product follows a separate route. IBM's documentation for the transform capabilities of watsonx Code Assistant for Z, which converts COBOL to Java, states that you can install the software in IBM Software Hub "if you want the entire solution to run on prem". IBM's installation guide for Software Hub 5.2.x lists GPU operators as always required for the watsonx Code Assistant products: watsonx Code Assistant for Z and its Code Explanation variant, and, from version 5.2.1, its Code Generation and Agentic variants. They are installed through the Node Feature Discovery Operator and the NVIDIA GPU Operator on Red Hat OpenShift. That page lists no GPU models, so confirm with IBM which card it supports before the hardware is ordered.

The two routes can run side by side. A team that licenses IBM's product for translation can still run an open model on its own GPU for explanation and documentation across all programs.

Hardware for each stage of the project

Sizing follows our users-per-card rule, as in our guide to how many users one RTX PRO 6000 serves: 90 per cent of the memory the driver reports, less 3 GiB per card, less the weights. gpt-oss-120b leaves 22.2 GiB on a 96 GB card, enough for 9 sessions of 64K tokens with a 16-bit cache, or 4 of 128K. The smallest RTX PRO card that holds it with useful context is therefore the RTX PRO 6000: on a 72 GB RTX PRO 5000 about 0.7 GiB would remain. Smaller models fit smaller cards, but in the study they trailed gpt-oss-120b on translation, the best of them, DeepSeekCoder-V2 16B, by 14 points, and mostly failed at writing COBOL; test one on your own programs before you plan on a smaller card. We count one working session per person at the busy hour and also cap each session at a planned 50 tokens per second, an upper bound by our estimate.

Sessions with gpt-oss-120b (65.3 GB, 60.8 GiB), 16-bit cache of 2.25 GiB per 64K by our arithmetic from its config.json, our users-per-card rule; planned is the smaller of memory and our 50 tokens-per-second ceiling, about 5 GB of active weights per token, 1,792 GB/s per Workstation Edition or Max-Q and 1,597 GB/s per Server Edition (NVIDIA). Estimates, not measurements.

Entry: one workstation. One RTX PRO 6000 Workstation Edition (600 W) or Max-Q (300 W) suits a team of three to nine people who explain and document programs and draft translations. Programs that need more than 64K tokens of context with their copybooks run in 128K sessions, which halves the count.

Two cards. A pair splits the model over two cards and leaves more than four times the cache room of one card, which suits a larger team or longer contexts. Two Workstation Edition cards are the practical maximum for that edition; three or four Max-Q belong only in a chassis built for them with strong directed airflow.

Servers. The inference node on our AI servers page carries four RTX PRO 6000 Server Edition cards with 768 GB of DDR5 ECC memory and draws about 3.4 kW by our estimate. It serves the team by day and runs documentation batches at night. Eight cards draw about 6.3 kW by our estimate, and Dell's datasheet gives about 8 kW per eight-card node as an indicative figure, so the feed is planned on the server maker's figure. An eight-card server also holds Qwen3-Coder-480B-A35B in FP8 (482 GB) with 13 sessions of 64K, 27 with an FP8 cache, for a team that wants to compare a larger coding model with gpt-oss-120b on its own code; eight H200 NVL hold it with 34. No COBOL result for that model is published; our guide to a server for coding agents has its sizing.

We build these workstations and AI servers to order; send us the number of programs, the people working on the project and the tasks you plan, and our team replies with a configuration.

A documentation pass over the whole code base

Documenting every program is a batch job: no one waits for a single answer, and what counts is whether the run finishes in a night. Our guide to batch LLM inference on-premise gives the method: count the programs, fix the output length per program, take a published throughput or time a sample, and divide.

For gpt-oss-120b on these cards there is a published figure. In MLPerf® Inference v6.1, an HPE ProLiant Compute DL380a Gen12 with eight RTX PRO 6000 Blackwell Server Edition cards produced 15,256 tokens per second on gpt-oss-120b in the offline scenario. As an example, 10,000 programs with 1,000 tokens of documentation each make 10 million output tokens; at that rate they take about 11 minutes by our arithmetic, about 22 minutes with the halving margin our batch guide applies until a sample has been timed. The benchmark uses its own inputs. A COBOL program with its copybooks is a much longer prompt, so reading the prompts takes a larger share of the time, and the rate on your code comes from a test run. The token count of your programs comes from the model's tokenizer, through the /tokenize endpoint that vLLM lists in its documentation. Even with a large margin, the GPU run is short next to the review of 10,000 drafts by your COBOL specialists.

If you plan a documentation run like this, ask us which configuration fits the size of your code base and your budget.

MLPerf® Inference v6.1, data centre, closed division, gpt-oss-120b, offline scenario, entry 6.1-0050, retrieved from mlcommons.org on 24 October 2026, result verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Governance in banks, insurers and public bodies

Running the model on your own hardware answers the first question of a security review: the source code and the business rules stay inside your network, and no third party processes them. The remaining questions are about the pipeline. Generated Java goes through the same review, static analysis and approval as any other code, and the characterisation tests are kept with it as evidence of equivalence. Prompts and answers contain source code, so decide before the first run whether the gateway stores them and for how long; our guide to an LLM gateway with keys and a query log covers the options. For banks and insurers, the rules of DORA, Regulation (EU) 2022/2554, on ICT risk and ICT third-party providers frame the platform, as our guide for that sector sets out; your legal counsel and your data-protection officer confirm how they apply to the project.

FAQ