Hacker News Reader

Comment by labrador | Hacker News Reader

I'm a retired programmer. I can't imagine trusting code generated by probablities for anything mission critical. If it were close and just needed minor tweaks I could understand that. But I don't have experience with it.

My comment is mainly to say LLMs are amazing in areas that are not coding, like brainstorming, blue sky thinking, filling in research details, asking questions that make me reflect. I treat the LLM like a thinking partner. It does make mistakes, but those can be caught easily by checking other sources, or even having another LLM review the conclusions.

View on HN

Well; I can't speak to your specific experience (current or past) but I'm telling you that while I'm skeptical as hell about EVERYTHING, it's blowing my expectations away in every conceivable way.

I built something in less than 24h that I'm sure would have taken us MONTHS to just get off the ground, let alone to the polished version it's at right now. The most impressive thing is that it can do all of the things that I absolutely can do, just faster. But the most impressive thing is that it can do all the things I cannot possibly do and would have had to hire up/contract out to accomplish--for far less money, time, and with faster iterations than if I had to communicate with another human being.

It's not perfect and it's incredibly frustrating at times (hardcoding values into the code when I have explicitly told it not to; outright lying that it made a particular fix, when it actually changed something else entirely unrelated), but it is a game changer IMO.

> I built something in less than 24h that I'm sure would have taken us MONTHS to just get off the ground, let alone to the polished version it's at right now

Would love to see it!

See, your comment is a good example of what's going wrong. The OP specifically mentioned "mission critical things" - My interpretation of that would be things that are not allowed to break, because otherwise people might die, in the worst case - and you were talking about just SOMETHING that got "done" faster. No mention about anything critical.

Of course, I was playing around with claude code, too, and I was fascinated how fun it can be and yes, you can get stuff done. But I have absolutely no clue what the code is doing and if there are some nasty mistakes. So it kinda worked, but I would not use that for anything "mission critical" (whatever this means).

> So it kinda worked, but I would not use that for anything "mission critical" (whatever this means).

It means projects like Cloudflare's new OAuth provider library. https://github.com/cloudflare/workers-oauth-provider

> "This library (including the schema documentation) was largely written with the help of Claude, the AI model by Anthropic. Claude's output was thoroughly reviewed by Cloudflare engineers with careful attention paid to security and compliance with standards. Many improvements were made on the initial output, mostly again by prompting Claude (and reviewing the results)."

https://neilmadden.blog/2025/06/06/a-look-at-cloudflares-ai-...

> It means projects like Cloudflare's new OAuth provider library.

the one that's a few weeks old and already has several CVEs due to the poor implementation?

The mission of a professional programmer is to deliver code that works according to the design specs, handles edge cases, fails gracefully and doesn't contain performance bottlenecks. It could be software for a water plant, or software that incurs charges to accomplish it's task and could bankrupt you if there is a mistake. It doesn't have to be a matter of life or death.

But there are a lot of projects and problem domains that don't even demand that much or have any real consequences for failure. I look at all the self-service stuff my employer has for HR, benefits, policy compliance, it's all half-broken, nobody ever seems to get held to fixing anything, and the only answers are "try it again later."

Professional programmers built this stuff too, or maybe it was vibe-coded but since it's been like that for years I think probably not.

But we don't know where on the spectrum of "people might die" to "try again later" most of these programmers who claim great productivity gains from LLMs lie. Maybe it is making them 10x faster at churning out shit, who knows? They might not even realise it themselves.

I don't understand. Of course you should read the code. There is no responsible engineer that would advocate for blindly accepting code written by another party for critical functions, be that a human or an LLM.

If you have absolutely no idea what the code is doing, something is wrong. Imagine someone with no engineering experience telling an engineer to build a jet engine "make it spin fast". Presented with the designs, he has no idea wether it's good. Obviously that is not workable. Rather, the one in charge for guiding, say, the design of a jet engine, would know the intricacies of the jet engine himself perfectly. It just would be too tedious to design all the little parts of it, and verify all the little characteristics by one person.

When developing with an LLM, you first figure out how you want to do it, and essentially start imagining the code. But rather than writing and testing, you tell the LLM enough to unambiguously implement what you have in mind, and how to test it/the expected behavour/scenarios you have in mind that it will support. Then you review the code; modern claude will not present you with buggy code, it will go through iterations itself to have something that works. The feedback than usually is something more about code style, consistency, or taking into account future expansions of the code.

We haven't even come up with different methodology yet to leverage these tools that well.

At some point one would have to think the idea of coming up with one design will seem outdated if the model is producing a 1000 different versions at once and then testing to find the best design. Then working on improving and tightening the design 24/7 365.

Most of what I read on here seems like knocking the automobile while proclaiming the virtues of the horse. It only makes sense because we can't see all the paved roads, gas stations and highways yet that make the horse a complete relic for travel.

At some point, it will make as much sense to pay a human to write code as it does to take a horse on the interstate.

I tried the "thinking partner" approach for a while and for a moment I thought it worked well, but at some point the cracks started to show and I called the bluff. LLMs are extremely good at creating an illusion that they know things and are capable of reasoning, but they really don't do a good job of cultivating intellectual conversation.

I think it's dangerously easy to get misled when trying to prod LLMs for knowledge, especially if it's a field you're new to. If you were using a regular search engine, you could look at the source website to determine the trustworthiness of its contents, but LLMs don't have that. The output can really be whatever, and I don't agree it's necessarily that easy to catch the mistakes.

You don't say what LLM you are using. I'm using ChatGPT 4o. I'm getting great results, but I review the output with a skeptical eye similar to how I read Wikipedia articles. Like Wikipedia, GPT 4o is great for surfacing new topics for research and does it quickly, which makes stream of thought easier.

This is very model-dependent. If you use something heavy on sycophancy and low on brain cells (like GPT-4o, the default paid ChatGPT model), you'll get lots and lots of cracks because these models are optimised for engagement.

That said, don't use model output directly. Use it to extract "shibboleth" keywords and acronyms in that domain, then search those up yourself with a classical search engine (or in a follow-up LLM query). You'll access a lot of new information that way, simply because you know how to surface it now.

Mostly agree. What works better is seeing the AI as a "high level to low level converter", in the same way Java is converted to machine code when it's ran. You describe exactly what you want it to report or do, and steer it whenever there are ambiguities. It does "grunt work" for you. With the bar of what grunt work means being moved up. Grunt work used to be doing the dishes or calculating numbers by hand on a paper spreasheet. Decades ago we automated those these. Now we've automated searching for information, summarization, implementing fully specified technical desings for software, the list goes on.

I've been programming for 40 years and have started using LLM's a few months ago, and it has really changed the way I work. I let it write pieces of code (pasting error messages from logs mostly result in a fix in less then a minute), but also brainstorming about architecture or new solutions. Of course I check the code it writes, but I'm still almost daily amazed at the intelligence and accuracy. (Very much unlike crypto).

That's good to know if I ever get an idea for a side project. Anything to relieve the tedius aspects of programming would be very welcome.

I say this as an LLM skeptic.

All code, including stuff that we experienced coders write is inherently probabilistic. That’s why we have code reviews, unit tests, pair programming, guidelines and guardrails in any critical project. If you’re using LLM output uncritically, you’re doing it wrong, but if you’re using _human_ output uncritically you’re doing it wrong too.

That said, they are not magic, and my fear is that people use copilots and agentic models and all the rest to hide poor engineering practice, building more and more boilerplate instead of refactoring or redesigning for efficiency or safety or any of the things that matter in the long run.

LLMs turn foot guns into foot bazookas. Still, if you learn to aim them away from your body they do make a bigger better boom.

There's one thing that I find LLM extremely good at: data science. Since the IO is well defined, you can easily verify that the output is correct. You can even ask it write tests for you given that you know certain properties of the data.

The problem is that the LLM needs context of what you are doing, contexts that you won't (or too lazy) to give in a chat with it ala ChatGPT. This is where Claude Code changes the game.

For example, you have PCAP file where each UDP packet contain multiple messages.

How do you filter the IP/port/protocol/time? Use LLM, check the output

How do you find the number of packets that have patterns A, AB, AAB, ABB.... Use LLM, check the output

How to create PCAPs that only contain those packets for testing? Use LLM, check the output

Etc etc

Since it can read your code, it is able to infer (because lets be honest, you work aint special) what you are trying to do at a much better rate. In any case, the fact that you can simply ask "Please write a unit test for all of the above functions" means that you can help it verify itself.

Treating it like a thinking partner is good. When programming, I don't treat it like a person, but rather a very high programming language. Imagine exactly how you want the code to be. Then find a way to express that unambiguously in natural language; the idea is that you'll still have a bit of work to do writing things out, but it will be a lot quicker than typing out all code by hand. Combined with iterations of feedback as well as having the AI build and run your program (at least as a sanity check); and asking the AI to check the program behaviour in the same way you would, gets you quite far.

A limitation is the lack of memory. If you steer it from style A to B using multiple points of feedback, if this is not written down, the next AI session you'll have to reexplain this all over.

Deepseek is about 1TB in weights; maybe that is why LLMs don't remember things across sessions yet. I think everybody can have their personal AI (hosted remote unless you own lots of compute); it should remember what happened yesterday; in particular the feedback it was given when developing. As an AI layman I do think this is the next step.

You don't trust the code coming out of the probabilistic machine. You build a validation cage around it with hard interfaces and you also review the output.

Reply on news.ycombinator.com