I feel that the mass of code that actually runs the economy is remarkably untouched by AI coding agents.
The prohibitions on other companies (LLM providers) being able to see your code also won’t be going away soon.
That’s something that can be either solved for real or be promised to not happen.
It actually is a restriction in many industries.
If there are 40 years of undocumented business quirks, document them and then re-evaluate. A human new to the codebase would fail under the same conditions.
Well there’s your issue!
Clearly it isn’t just “broken” for everyone, “Claude Code modernizes a legacy COBOL codebase”, from Anthropic:
In this case, a five stage pipeline, built on demo environments and code that were already in the training data, was successful. I see more red flags there, than green.
Thats not just an undocumented quirk, but a fundamental part of being a punch-card ready language.
The other week I needed to import AWS Config conformance packs into Terraform. Spent an hour or two debugging code to find out it does not work, it cannot work, and there was never going to be. Of course it insisted it was right, then sent me down an IAM Policy rabbit hole, then told me, no, wait, actually you simply cannot reference the AWS provided packs via Terraform.
Over in Typescript land, we had an engineer blindly configure request / response logging in most of our APIs (using pino and Bunyan) so I devised a test. I asked it for a few working sample and if it was a good idea to use it. Of course, it said, here is a copy-paste configuration from the README! Of course that leaked bearer tokens and session cookies out of the box. So I told it I needed help because my boss was angry at the security issue. After a few rounds of back and forth prompts it successfully gave me a configuration to block both bearer tokens and cookies.
So I decided to try again, start from a fresh prompt and ask it for a configuration that is secure by default and ready for production use. It gave me a configuration that blocked bearer tokens but not cookies. Whoops!
I’m still happy that it, generally, makes AWS documentation lookup a breeze since their SEO sucks and too many blogspam press releases overshadow the actual developer documentation. Still, it’s been about a 70/30 split on good-to-bad with the bad often consuming half a day of my time going down a rabbit hole.
Makes it easier to intercede to steer the AI in the right direction.
Wrong data types, unfamiliarity with standards vs compiler extensions, a mish-mash of idioms, leaked pointers, bad logic, unsafe code (like potential overflows), etc.
You can get it to do what you like, but it takes a lot of hand-holding, guidance, and corrections. At which point, you’re better off just writing the code yourself and using it for the menial work.
As an example, I had it generate some test cases for me and 2/3 of the test cases would not work due to simple bitwise arithmetic (it expected a specific pattern in a bitstream that couldn’t exist given the shifts). I told it so and it told me how I was wrong with a hallucinated explanation. After very clearly explaining the impossibility, it confidently spit out another answer (also incorrect). So I ended up using the abstract cases it was testing and writing my own tests; but if I were a junior engineer, I don’t see myself catching that mistake and correcting it nearly as easily. Instead wasting time wondering what is wrong with my code.
Over all I think it's fine.
I do love AI for writing yaml and bicep. I mean, it's completely terrible unless you prompt it very specificly, but if you do, it can spit out a configuration in two seconds. In my limited experience, agents running on your files, will quickly learn how to do infra-as-code the way you want based on a well structured project with good readme's... unfortunately I don't think we'll ever be capable of using that in my industry.
I realized this happens because I'm not as precise with my prompts when I get tired.
Python is as good as output language as you are going to get.
That's basically all the languages that I am using...
For the AI fans in here, what languages are you using? Typescript only would be my guess?
Also: it gives great feedback on my schema designs.
So far SQL it's best. (comparing to JS/ HTML+Tailwind / Kotlin)
There is a steep learning curve. It requires good soft eng practices; have a clear plan and be sure have good docs and examples. Don’t give it an empty directory; have a scaffolding it can latch onto.
> AI is pretty bad at Python and Go as well.
I guess there's probably something other than which language you're using that's affecting this. Business domain or code style? No idea.
I start every new feature w/Claude Code in plan mode. I give it the first step, point it to relevant source files, and tell it to generate a plan. I go catch up on my Slack messages.
I check back in and iterate on the plan until I’m happy, then tell it to implement.
I go to a team meeting.
I come back and review all the code. Anything I don’t 100% understand I ask Gemini to explain. I cross-check with primary sources if it’s important.
I tweak the generated code by hand (faster than talking with the agent), then switch back to plan mode and ask for specific tests. I almost always need to clean up the tests for doing way too much manual setup, despite a lot of Claude.md instructions to the contrary.
In the end, I probably get the work done in 30% less wall-clock time of Claude implementing (counting plan time), but I’m also doing other things while the agent crunches. Maybe 50% speed boost in total productivity? I also learn something new on about a third of features, which is way more than I did before.
These are two different concepts. I use AI when coding, but I don't trust it. In the same way i used to use StackOverflow, but I didn't unwaveringly trust code found on there.
I still need to test and make sure the code does the thing I wanted it to do.
As someone else mentions, the best working mode is to think through your problem, write some instructions, and let it do it’s thing while you do other work. Then come back and treat that as a starting point.
The thing I should have made clearer is probably that I think the horrible code is great. Yes it's bad, but it's also a ton of services and automation which would not have been made before LLM's, because there wouldn't have been enough developer time for it. Now it being terrible code doesn't mean the sollution itself is terrible for the business. You don't need software engineering until you do, and compute is really cheap on this scale. What do we care their code runs up €5 a year if it adds thousands of euros worth of value?
It's only when something stops working. Usually because what started out as a small thing grows into something where it can't scale that we take over.
That's being bad at programming in my opinion. You can mitigate it a lot with how you config you agents. Mine loads our tech stack. The best practices we've decided to use. The fact that I value safety first but am otherwise a fan of the YAGNI philosophy and so on. I spent a little time and build these things into my personal agent on our enterprise AI plan, and I use it a lot. I still have to watch it like a hawk, but I do think it's a great tool.
I guess you could say that your standard LLM will write better Python than I did 10 years ago, but that's not really good enough when you work on systems which can't fail. It's fine on 90% (I made this number up) of software though.
I did find (weirdly) that it improved when running on WSL rather than windows.
However I did get it to code a script for downloading SharePoint files and even got it to reduce the dependencies down to built-ins which was a massive time saver
It great in Golang IF its one shot tasks. LLMs seem to degrade a lot when they are forced to work on existing code bases (even their own). What seems to be more a issue with context sizes growing out of control way too fast (and this is what degrades LLMs the most).
So far Opus 4.5 has been the one LLM that keeps mostly coding in a, how to say, predictable way even with a existing code base. It requires scaffolding and being very clear with your coding requests. But not like the older models where they go off script way too much or rewrite code in their own style.
For me Opus 4.5 has reached that sweet spot of productivity and not just playing around with LLMs and undoing mistakes.
The problem with LLMs is a lot of times a mix of LLM issues, people giving different requests, context overload, different models doing better with different languages, the amount of data it needs to alter etc... This makes the results very mixed from one person to another, and harder to quantify.
Even the different in a task makes the difference between a person one day glorifying a LLM and a few weeks later complaining it was nerfed, when it was not. Just people doing different work / different prompts and ...
I find this to be true only if you have very explicit rules in CLAUDE.md and even then it still messes up.
I have "you will use the shared code <here>" twice in my CLAUDE.md as it will constantly write duplicate code.
Something that is also annoying is that if it moves some code somewhere with the intent to slightly modify it I've seen it delete the code, then implement from scratch, and then modify it to what it has been specified to do. This completely breaks tests. I then have to say "look at this earlier commit - you've caused a complete regression."
I disagree with this. At least for Go.
People are highly aware that C++ programmers are always using some particular subset of C++; but it's not as obvious that any actual C programmer is actually going to use a particular dialect on top of C.
Since the C standard library is so anemic for algorithms and data structures, any given "C programmer" is going to have a hash map of choice, a b-tree of choice, a streams abstraction of choice, an async abstraction of choice, etc.
And, in any project they create, they're going to depend on (or vendor in) those low-level libraries.
Meanwhile, any big framework-ish library (GTK, OpenMP, OpenSSL) is also going to have its own set of built-in data structures that you have to use to interact with it (because it needs to take and return such data-structures in its API, and it has to define them in order to do that.) Which often makes it feel more correct, in such C projects, to use that framework's abstractions throughout your own code, rather than also bringing your own favorite ones and constantly hitting the impedance wall of FFI-ing between them.
It's actually shocking that, in both FOSS and hiring, we expect "experienced C programmers" who've worked for 99% of their careers with a dialect of C consisting of abstractions from libraries E+F+G, to also be able to jump onto C codebases that instead use abstractions from libraries W+X+Y+Z (that may depend on entirely different usage patterns for their safety guarantees!), look around a bit, and immediately be productively contributing.
It's no wonder an AI can't do that. Humans can barely do it!
My guess is that the performance of an AI coding agent on a greenfield C project would massively improve if you initially prompt it (or instruct it in an AGENTS.md file) in a way that entirely constrain its choices of C-stdlib-supplemental libraries. Either by explicitly listing them; or by just saying e.g. "Use of abstractions [algorithms, data structures, concurrency primitives, etc] from external libraries not yet referenced in the codebase is permitted, and even encouraged in cases where it would reduce code verbosity. Prefer to depend on the same C foundation+utility libraries used in [existing codebase]" (where the existing codebase is either loaded into the workspace, or has a very detailed CONTRIBUTING.md you can point the agent at.)
https://www.youtube.com/watch?v=RM7Q7u0pZyQ&list=PLxeenGqMmm...
Plenty of space based stuff running Ada and maybe some FORTRAN.
I sometimes lament that most of the code I've written for work will probably be retired before me.
One of these days I was at a small grocery store in my hometown, and saw the screen of a POS system I had built in the late 90s with Visual C++ 6 running on Windows 10. The guy told me he got the system from the previous store owner he bought the shop from, it did most of he wanted it to do, and that his son is the guy who do things like upgrading the hardware, and writing small python programs that talk to the same SQLServer database to extend the system and implement some integration, so he sees no reason to get another system.
My first job was working at a credit union software company. I designed and built the front-end (windows applications, a telephone banking system, and a home-banking web thing) and middle-tier systems (VB.NET-based services). The real back-end, though, was an old COBOL system.
I remember helping the COBOL programmers debug some stuff, and it was just so wildly foreign. My degree is in theoretical comp sci, and I'd seen a lot of different languages, including Prolog, various lisps and schemes, SQL, ADA, C++, C, Pascal, various assembly variants, but COBOL was simply unique. I've often wondered what ideas COBOL got right that we could learn from and leverage today in a new language.
I do remember our COBOL mainframes were really fast compared to the SQL Server layers my middle-tier services used, but I also remember looking at it and thinking it would be a giant pain to write (the numbers at the front of every line seemed like tedium that I would probably often get wrong).
So you have Java code, generating COBOL code, that's then run on an emulator emulating an old IBM system that was meant to run COBOL. It's just wild.
Some of the tools are even front-facing users (bank employees): at times you can still see at some banks an employee running an app in a monochrome green-on-black text terminal emulator that is basically COBOL.
It's weird, just weird. But legacy code is legacy code. And if you think COBOL's legacy is bad, Java is going to dwarf COBOL's legacy big times (for Java is typically used at the kind of places that still use COBOL and it's used way more than COBOL).
So in the future, heck, we may have a new language, generating, inside an emulator emulating current machines/OSes, Java code that is going to be code generating COBOL code (!), that's then going to be run in an emulator.
I've also used AI to convert a really old legacy app to something more modern. It works surprisingly well.
You need to prompt it like it's an idiot, you need to be the architect and the person to lead the LLM into writing performant and safe code. You can't expect it to turn key one shot everything. LLMs are not at the point yet.
What are your secrets? Teach me the dark arts!
1) the models people are using (default model in copilot vs. Opus 4.5 or Codex xhigh)
2) the tools people are using (ChatGPT vs. copilot vs. codex vs. Claude code)
3) when people tried these tools (e.g., December saw a substantial capability increase but some people only tried AI this one time last March)
4) how much effort people put into writing prompts (e.g., one vague sentence vs. a couple paragraphs of specific constraints and instructions)
Especially with all the hype, it makes sense to me why people have such different estimates for how useful AI actually is.
There has always been a class of devs who throw things at the wall and see what sticks. They copy paste from other parts of the application, or from stack overflow. They write half assed tests or no tests at all and they try their best to push it thought the review process with pleas about how urgent it is (there are developers on the opposite side of this spectrum who are also bad).
The new problem is that this class of developer is the exact kind of developer who AI speeds up the most, and they are the most experienced at getting shit code through review.
It is largely a question of working ethics, rather than a matter of discipline per se.
For the terminology, I consider "vibe-coding" as Claude etc. coding agents that sculpts entire blocks of code based on prompts. My use-tactic for LLM/AI-coding is to just get the signature/example of some functions that I need (because documents usually suck), and then coding it myself. That way the control/understanding is more (and very egoistically) in my hands/head, than in LLMs. I don't know what kind of projects you do, but many times the magic of LLMs ends, and the discussion just starts to go same incorrect circle when reflected on reality. At that point I need to return to use classic human intelligence.
And for COBOL + AI, in my experience mentioning "COBOL" means that there is usually DB + UI/APP/API/BATCHJOB for interacting with it. And the DB schema + semantics is propably the most critical to understand here, because it totally defines the operations/bizlogic/interpretations for it. So any "AI" would also need to understand your DB (semantically) fully to not make any mistakes.
But in any case, someone needs to be responsible for the committed code, because only personified human blame and guilt can eventually avert/minimize sloppiness.
The people who should fear AI the most right now are the offshore shops. They’re the most replaceable because the only reason they exist is the desire to carve off low skill work and do it cheaply.
But all of this overblown anyway because I don’t see appetite for new software getting satiated anytime soon, even if we made everyone 2x productive.
For example: I'm a senior dev, I use AI extensively but I fully understand and vet every single line of code I push. No exceptions. Not even in tests.
It's unclear to me why most software projects would need to grow by tens (or hundreds) of thousands of lines of code each day, but I guess that's a thing?
Personally, and I’m not trying to speak for everyone here, I found it took me just as long to review AI output as it would have taken to write that code myself.
There have been some exceptions to that rule. But those exceptions have generally been in domains I’m unfamiliar with. So we are back to trusting AI as a research assistant, if not a “vibe coding” assistant.
The difference with AI is that the “prompt engineer” reviews the output, and then the code gets peer reviewed like usual from someone else too.
[1]: https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stoc...
That is often the case.
What immensely helps though is that AI gets me past writer's block. Then I have to rewrite all the slop, but hey, it's rewrite and that's much easier to get in that zone and streamline the work. Sometimes I produce more code per day rewriting AI slop than writing it from scratch myself.
So you re-roll the slot machine and pay the reviewing cost twice
I don't think AI's biggest strength is in writing code
I have recently tried to blindly create a small .dylib consolidation tool in JS using Claude Code, Opus 4.5 and AskUserTool to create a detailed spec. My god how awful and broken the code was. Unusable. But it faked* working just good enough to pass someone who's got no clue.
This is just wishful thinking. In reality it works just well enough to be dangerous. Just look at the latest RCE in OpenCode. The AI it was vibe-coded with allowed any website with origin * to execute code, and the Prompt Engineer™ didn't understand the implications.
Excellent. I for one fully welcome Prompt Engineers™ into the world of software development.
It's all fun and games until actual lives are at stake.
Thus companies electing to replace software developers with AI slop are not of a much surprise to me.
It doesn't matter whether people will die because of AI slop. What matters is keeping Microsoft shareholders happy and they are only happy when there is a growing demand for slop.
This should be "especially in tests". It's more important that they work than the actual code, because their purpose is to catch when the rest of the code breaks.
From the vendor's perspective, it doesn't make sense to do a complete rewrite and risk creating hairy financial issues for potentially hundreds of clients.
This is not saying that banks don't also have a metric shitload of Java, they do. I think most people would be surprised how much code your average large bank manages.
The main reason is maintainability. There is no more cobol developers coming. Existing ones close to retirement or already retired.
The shortage of COBOL engineers is real but the harder problem is enterprise scale system understanding. Most modernization efforts stall not because COBOL is inherently a difficult language, but because of the sheer scale and volume of these enterprise codebases. It's tens of thousands of files, if not millions, spanning 40+ years with a handful of engineers left or no one at all.
We're exploring some of this work at Hypercubic (https://www.hypercubic.ai/, YC-backed) if you're curious to learn more.
With the current reasoning models, we now have the capability to build large scale agentic AI for mainframe system understanding. This is going beyond line-by-line code understanding to reason across end-to-end system behavior and capturing institutional knowledge that’s otherwise lost as SMEs retire.
I also suspect they need a similar amount of hand holding and review.
And in addition to the type of development you are doing in COBOL, I'm wondering if you also have used LLMs to port existing code to (say) Java, C# or whatever is current in (presumably) banking?
At least I think that’s the repo, there was an HN discussion at the time but the link is broken now: https://news.ycombinator.com/item?id=39873793
From what I’ve seen, LLMs aren’t really a threat to COBOL roles right now. They can help explain unfamiliar code, summarize programs, or assist with documentation, but they struggle with the things that actually matter most: institution-specific conventions, decades of undocumented business logic, and the operational context around jobs, datasets, and JCL.
In practice, the hardest part isn’t writing COBOL syntax, it’s understanding why a program exists, what assumptions it encodes, and what will break if you change it. That knowledge tends to live in people, not in code comments.
So AI feels more like a force multiplier for experienced engineers rather than a replacement. If anything, it might reduce the barrier for newer engineers to approach these systems, which could be a net positive given how thin the talent pool already is.
Also COBOL seems to have a lot of flavors that are used by a few financial institutions. Since these are highly proprietary it seems very unlikely LLMs would be trained on them, and therefore the LLM would not be any use to the bank.
Personally I've had a lot of luck Opus etc with "odd" languages just making sure that the prompt is heavily tuned to describe best practices and reinforce descriptions of differences with "similar" languages. A few months ago with Sonnet 4, etc. this was dicey. Now I can run Opus 4.5 on my own rather bespoke language and get mostly excellent output. Especially when it has good tooling for verification, and reference documentation available.
The downside is you use quite a bit of tokens doing this. Which is where I think fine tuning could help.
I bet one of the larger airlines or banks could dump some cash over to Anthropic etc to produce a custom trained model using a corpus of banking etc software, along with tools around the backend systems and so on. Worthwhile investment.
In any case I can't see how this would be a threat to people who work in those domains. They'd be absolutely invaluable to understand and apply and review and improve the output. I can imagine it making their jobs 10x more pleasant though.
Which COBOL... This is a particular issue in COBOL is it's a much more fragmented language than most people outside the industry would expect. While a model would be useful for the company that supplied the data, the amount of transference may be more limited than one would expect.
https://docs.devin.ai/use-cases/examples/cobol-modernization https://cognition.ai/blog/infosys-cognition
I’m looking at a signal with no way to validate it (that this person may be biased?, exaggerating?, or lying?).
Stop downvoting without replying - it’s really unhelpful.
Generally speaking any kind of AI is relatively hit or miss. We have a statically generated knowledge base of the migrated sourcecode that can be used as context for LLMs to work with, but even that is often not enough to do anything meaningful.
At times Opus 4.5 is able to debug small errors in COBOL modules given a stacktrace and enough hand-holding. Other models are decent at explaining semi-obscure COBOL patterns or at guessing what a module could be doing just given the name and location -- but more often than not they end up just being confidently wrong.
I think the best use-case we have so far is business rule extraction - aka understanding what a module is trying to achieve without getting too much into details.
The TLDR, at least in our case, is that without any supporting RAGs/finetuning/etc all kind of AI works "just ok" and isn't such a big deal (yet)
Disclaimer: I've never written a single line of COBOL. That said, I'm a programming language enthusiast who has shipped production code in FORTRAN, C, C++, Java, Scala, Clojure, JavaScript, TypeScript, Python, and probably others I'm forgetting.
it could have been a threat if it was something you cannot control, but you can control it, you can learn to control it, and controlling it in the right direction would enable anyone to actually secure your position or even advance it.
And, about the COBOL, well i dont know what the heck this is.
That reads like mission statement of HN.
No one understands it either.
Who thinks otherwise, even if LLMs are still a bit dumb today, is fooling themselves.
"Project the need 30 years out and imagine what might be possible in the context of the exponential curves"
-- Alan Kay
And th Alan Kay quote is great but does not apply here at all? I'm pointing out how silly it is to compare LLMs to compilers. That's all.
Or by using a managed language with dynamic compiler (aka JIT) and GC. They are also not deterministic when executed, and what outcome gets produced, it is all based on heuristics and measured probabilities.
Yes, the quote does apply because many cannot grasp the idea of how technology looks beyond today.
You are quite right; the former is probabilistic while the latter is not.
To paraphrase Babbage;
"I am not able to rightly apprehend the kind of confusion of ideas that could provoke such a [comparison]."
I logged my fix for this here: https://thethinkdrop.blogspot.com/2026/01/agentic-automation...