Hacker News Reader

Comment by toplinesoftsys | Hacker News Reader

The biggest problem is not that bugs are migrated with COBOL, but that lots of new bugs are going to be introduced. AI is not deterministic, it will be making tons of mistakes. The only realistic low-error approach is incremental step-by-step migration using Cursor or similar tools. However, it requires much more time as each step must be prompted, tested and committed manually. Any hope that one-shot migraton of a large code base will not introduce enormous number of bugs is very naive. LLM is very bad on handling long context - it is their nature unfortunately. There is no answer to this problem yet.

View on HN

> AI is not deterministic, it will be making tons of mistakes.

From the paper:

> The COBOL source is passed through an internal deterministic Migrator to produce a generated Java target.

Also, humans are not deterministic either. Give the same COBOL -> Java translation to multiple developers and each will come up with a different solution. Heck, even the same developer will produce a different output for the same task, depending on the day of the week.

> Also, humans are not deterministic either.

Indeed, and that's why so few dare to migrate them, and so many who do fail or blow the budget many times over.

I think that was the implicit point of the comment: don't expect that with AI, suddenly we can convert all those COBOL apps with a single prompt.

But it says that the agentic authoring step patches the migrator when things get stuck. So while the execution of this stuff is deterministic, its actual content is not.

> Also, humans are not deterministic either.

And that's why it's usually a stupid idea to nilly-willy migrate large code bases to different languages (also I'm getting really tired of the "but humans aren't either" trope).

Which is why N-version programming is often used for systems that need to work (like plane software) and even then, as we know today, it can not be perfect. Google "plane software bugs history" for an insight into that.

So they solved the "make no mistakes" problem? They deserve a nobel prize.

Deterministic != Correct

> AI is not deterministic, it will be making tons of mistakes.

Just set the sampling temperature to zero and remove any unintended non-determinism during the parallel computation of the token probability distribution. The problem is solved? Of course, not. Non-determinism has little to do with LLMs' mistakes.

If a considerable size of test data is available for any system, rewriting a well-understood one is much easier today than rewriting by hand. The issue is mostly with the “well-understood” part, as over the years, none cared about understanding and it became a working blackbox that is responsible for a critical part of large system and none wants to take the risk.

> AI is not deterministic, it will be making tons of mistakes.

Most compilers are also not deterministic, at least by default, but they don't usually make mistakes. Determinism isn't an important quality here. And if it were, AI can be deterministic, it just isn't normally for much the same reason compilers typically aren't (hint: performance).

I find it bizarre that I keep reading this here. Just one of those things that keeps getting blindly repeated without receiving any thought?

They cannot be made entirely deterministic. You have cases where the next set of token choices have the same probability, and most people also are using cloud provider served models which can be served across different hardware implementations which can modify the math used in running the model.

can you get some arbitrary number of 9s for consistency? Yea. You can’t get 100% deterministic though.

There is nothing about LLMs that prevent them from being 100% deterministic, unless you are really stretching the definition. There are implementation shortcuts often used in practice, just as is the case with compilers, that lead to non-determinism but there is no reason you have to rely on those implementation details. Not even if cloud providers do. LLMs can be run in environments under your control.

Computers are deterministic. When we talk about non-determinism we're just talking about where there are hidden inputs, but the fundamentals of computing means that all hidden inputs can become visible if you want them to be (although possibly at the cost of things like performance), so, yes, you can reach 100% determinism just fine if you wish to. It's just not worth the tradeoffs in most cases. But if a deterministic solution solved a problem here it would be worth it.

However, a deterministic LLM doesn't help here. LLMs don't "make mistakes" because they are typically non-deterministic. They would equally "make mistakes" when deterministic.

You don’t need the entire codebase in context in every moment to migrate it. Also AI being non deterministic does not prevent it from one-shotting perfect solutions 100% of the time for simple enough problems. And every model generation brings this bar higher. So that’s really not a fundamental problem. And we can also implement llm inference deterministically if we want, it’s just that it’s not worth the loss in performance to do it.

> And we can also implement llm inference deterministically if we want, it’s just that it’s not worth the loss in performance to do it.

My understanding is thats not possible (different from being practical), wonder if you have any literature, research to back up that claim?

This one from thinking machines: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... Was discussed a lot also here on hn

Thanks for sharing, it was really good read. The authors in the article have done a good job for sure to validate their hypothesis and make it work for a particular subset of problem.

1. Local hardware, no networking or HTTP requests 2. No other parallel requests 3. 1000 runs bounded by 1000 tokens

I have worked a little bit in academia and I am not a big fan of the way favorable samples for the hypothesis are kept and unfavorable ones are thrown away. I might be wrong here, however its highly unlikely that the team would have just worked with one query. Chances are that a lot of different prompts with varying number of runs would have been tried to see what works and supports the hypothesis.

> The only realistic low-error approach is incremental step-by-step migration using Cursor or similar tools.

Correction: AI is not deterministic, the only realistic low-error solution is not a more complex use of non-deterministic AI, but deterministic transpilation.

The problem is that this results in COBOL-in-Java which runs correctly but it is a nightmare to maintain.

AI (LLMs) is typically not run deterministically for performance reasons, but can be turned deterministic if you want it to be. The output will still be probabilistic, however, so you will be no further ahead. Determinism isn't helpful.

You are, of course, correct about determinism. In the grandparent post I was accepting the previous poster’s somewhat sloppy use of “nondeterministic” because I was focussed on other aspects of the discussion.

AI doesn't need to be deterministic.

But, hear me out, a COBOL to Java migration absolutely needs to be deterministic. Surviving COBOL programs run a lot of sensitive operations.

As a side note, I can't fathom why they chose Java. It's easier to teach COBOL to a competent engineer than to rewrite everything in a language that encourages onion architectures. Why not Go, for example?

Java as a programming language doesn't enforce any architecture. There is no difference to go. Its a system of folders.

The tests need to be green, the math needs to math, but the code can look a little bit different here or there.

I don't think its easy at all to educate someone on COBOL. Its carrier suicide if you want to ever do anything else again after that gig. I was writing java for years, switched for 2 years to php for another company and had a few companies complaining about these 2 years...

As someone that has worked at a bank maintaining RPG/COBOL and Java, the latter definitely encourages programmers to wrap things in yet another class. They think they're simplifying, but in reality, they're piling abstraction layers that have to be peeled off one by one when, for example, you're tring to find that EBCDIC-to-UTF8 bug.

A good Java architecture is possible, though, but it requires planning, discipline, and restraint; and yet the ever-changing business needs can mar your beautiful castle given enough time. COBOL is simpler and more straightforward, you have to go out of your way to mess it as badly as Java allows.

Does it? I found llms to be great for straight conversions

At least if tescoverage is good, but well... That's something llms can also be used for

Reply on news.ycombinator.com