Hacker News Reader

Comment by johnfn | Hacker News Reader

The article throws out a lot of potential issues with AI generated code, but doesn't stop for a moment to consider if solutions currently exist or might exist to these problems.

- Before LLMs, provided that your team did some of their due diligence, you could always expect to have some help when tackling new code-bases.

Has the author never worked on legacy code before?

- (oh, and it has forgotten everything about the initial writing process by then).

Does the author not think this can ever be solved?

- Because of the situation of a Bus Factor of zero that it creates, vibe coding is fundamentally flawed. That is, only until there is an AI that can generate 100% accurate code 100% of the time, and it is fed 100% accurate prompts.

Why does it require 100% accuracy 100% of the time? Humans are not 100% accurate 100% of the time and we seem to trust them with our code.

View on HN

Yea, I found the article to be overly reductive. I work on a shared ownership codebase, which at times, I am not going to be able to pull the original author at all times when I work on my branch.

At least having a partner explain some aspects is a huge unlock for me. Maybe the OP shadowboxing a world with no humans, but no humans is already a status quo issue that my team and I face sometimes.

He’ll, have they never had to maintain applications where the source code was lost? I haven’t had to do it too often, and I wouldn’t claim I’m very good at it, but on more than a handful of occasions I have had to decompile binaries to figure out what the fuck it was doing before I could write a wrapper around it to try and mitigate its unfixeable issues

The author was just saying that this will become the norm, not the exception. Ok, not this bad -- we can at least expect to have access to the AI-generated code for at least a little while longer. (I can imagine a future where AI programming gets just robust enough that some early adopter types will take their whiteboard sketches and prompts and prose test case descriptions, generate the code, compile it to a binary, then throw out the code. It's more agile! You can tweak the spec slightly and just turn the crank again!)

Hello, author here. Thanks for your comment.

I agree with your first point, maybe AI will close some of those gaps with future advances, but I think a large part of the damage will have been done by then.

Regarding the memory of reasoning from LLMs, I think the issue is that even if you can solve it in the future, you already have code for which you've lost the artifacts associated with the original generation. Overall I find there's a lot of talks (especially in the mainstream media) about AI "always learning" when they don't actually learn new anything until a new model is released.

> Why does it require 100% accuracy 100% of the time? Humans are not 100% accurate 100% of the time and we seem to trust them with our code.

Correct, but humans writing code don't lead to a Bus Factor of 0, so it's easier to go back, understand what is wrong and address it.

If the other gaps mentioned above are addressed, then I agree that this also partially goes away.

How is that any different than any other legacy code where the reason for the decisions being made have long been forgotten?

> Regarding the memory of reasoning from LLMs, I think the issue is that even if you can solve it in the future, you already have code for which you've lost the artifacts associated with the original generation. Overall I find there's a lot of talks (especially in the mainstream media) about AI "always learning" when they don't actually learn new anything until a new model is released.

But this already exists! At work, our code is full of code where the original reasoning for the code is lost. Sometimes someone has forgotten, sometimes the person who wrote it is no longer at the company any more, and so on.

> Correct, but humans writing code don't lead to a Bus Factor of 0, so it's easier to go back, understand what is wrong and address it.

But there are plenty of instances where I work with code that has a bus factor of 0.

The conclusion of your article is that vibe coding is "fundamentally flawed". But every aspect you've described about vibe coding has an analog in normal software engineering, and I don't think you would claim that is "fundamentally flawed".

I can't speak for the author, but I would definitely claim that having a bus factor of zero for any remotely-mission-critical piece of software is "fundamentally flawed", no matter the cause. I'd say the same for a bus factor of one in most settings.

I think that's moving goalposts. The original post never talks about vibe-coding mission-critical software - and I wouldn't advocate for that, either. The post says that all vibe coding is fundamentally flawed.

That's fair, and I agree with you that the generalizations in the article's conclusion go too far.

I added the "remotely-mission-critical" qualifier to capture additional nuance. Tolerance for a low bus factor should be inversely correlated with a project's importance. That wasn't explicitly stated in the article, but it seems uncontroversial, and I suspect the author would agree with me.

To make sure I understand your position:

> But there are plenty of instances where I work with code that has a bus factor of 0.

Do you think this is a problem?

As per my other replies, if all of these instances are in completely unimportant projects, then I could see you answering "no" (but I'd be concerned if you're spending a lot of time on unimportant things). If they are important, isn't the fact that knowledge about them has been lost indicative of a flaw in how your team/company operates?

I do think this is a problem. But the article goes one step further than claiming it's a problem - it claims that a bus factor of 0, and by extension, vibe coding, is "fundamentally flawed". I don't think that a bus factor of 0 is indicative of a "fundamental flaw".

I think they meant:

1. If a process necessarily results in bus factors of zero, that process is flawed.

2. The nature of vibe coding is such that it always produces code with a bus factor of zero (i.e. this is a "fundamental" fact of vibe coding).

I definitely agree with the first point, and I think agree with the second as well (at least if "vibe coding" carries its original implication that you don't even look at/care about the code produced by the LLM).

Did you have a different interpretation? Or do you disagree with one of these points?

Thanks for breaking that down.

I think I would disagree a bit with both.

It doesn't seem a priori evident that producing a bus factor of zero is bad. For instance, using a library which is not maintained has a bus factor of zero. That doesn't seem "flawed" to me. I think the problem I find with the author's, and your, statements is absolutism. Few things in engineering are ever truly 100% right or 100% wrong choices. Using a library with no maintainers is decision with tradeoffs, but in some contexts those tradeoffs would be worth it. Similarly, vibe coding code that has a bus factor of zero is also a decision with tradeoffs, and sometimes tradeoffs are worth it.

As for the second point - is it really so. hard to read the code that an LLM produces? I am continuously reading output from LLMs for any code which is even remotely important. Again, this is another decision with tradeoffs. Sometimes I vibe-code a 500 loc script, but I can manually verify the output of the script, and therefore there is no reason to read every line. Sometimes I am working on more important code that must be right; I typically inspect it line by line, like a code review.

Why would the damage have been done by then? What does that even mean?

The solution is almost trivial and well known:

Document. Whenever a member of the team leaves, you should always make them document what is in their head and not already documented (or even better, make documenting a required part of normal development).

The thing to understand is that every (new) AI chat session is effectively a member entering and leaving your team. So to make that work well you need great onboarding (provide documentation) and great offboarding (require documentation).

I _wish_ I had an AI tool when digging into some legacy codebases in my past.

"Just ask the author of that perl file".

Sure, it was last edited in -97 and the author is the regional manager now, should I just book a meeting or...?

> Why does it require 100% accuracy 100% of the time?

I've said this before, but I'd say it again: anti-AI people, instead of AI users, are usually those who expect AI to be magical panacea.

The vibe reminds me of some people who are against static typing because "it can't catch logical error anyway."

No, we treat it as a bureaucratic, distracting and infantile way of software engineering that does not work.

No, it's not a wish nor an expectation for a magical panacea, but logic.

The only code you do not have to maintain is code that is fully perfect (100% accurate, 100% of the time). Bus factor zero is a problem for code that needs to be maintained. If AI is short of the 100%, then it is generating code that needs to be maintained. Ergo, a bus factor zero will be a problem for AI generated code (until such time as that code is perfect).

Reply on news.ycombinator.com