Hacker News Reader

Comment by alganet | Hacker News Reader

> here are some example ideas that are perfectly true when applied to regular software

Hm, I'm listening, let's see.

> Software vulnerabilities are caused by mistakes in the code

That's not exactly true. In regular software, the code can be fine and you can still end up with vulnerabilities. The platform in which the code is deployed could be vulnerable, or the way it is installed make it vulnerable, and so on.

> Bugs in the code can be found by carefully analysing the code

Once again, not exactly true. Have you ever tried understanding concurrent code just by reading it? Some bugs in regular software hide in places that human minds cannot probe.

> Once a bug is fixed, it won’t come back again

Ok, I'm starting to feel this is a troll post. This guy can't be serious.

> If you give specifications beforehand, you can get software that meets those specifications

Have you read The Mythical Man-Month?

View on HN

You should read the footnote marked [1] after "a note for technical folk" at the beginning of the article. He is very consciously making sweeping generalizations about how software works in order to make things intelligible to non-technical readers.

Sure, but:

> these claims mostly hold, but they break down when applied to distributed systems, parallel code, or complex interactions between software systems and human processes

The claims the GP quoted DON’T mostly hold, they’re just plain wrong. At least the last two, anyway.

Say more? I stand by my statement, but you're not specific enough for me to explain why I believe I'm correct.

Focusing on the last two, which I called out specifically:

> Once a bug is fixed, it won’t come back again

Regressions are extremely common, which is why regression tests are so important. It is definitely not uncommon for bugs that were fixed once to come back again. This statement doesn't "mostly hold".

> If you give specifications beforehand, you can get software that meets those specifications

In theory, maybe, but in practice its messy. It depends on your acceptance testing. It depends on whether stakeholders change their mind during implementation (not uncommon). It depends on whether the specification was complete and nothing new is learned during development (almost never the case). Providing a specification in advance does not necessarily mean what you get out the other end meets that specification, unless its a relatively small, trivial, non-changing piece and the stakeholders have the discipline to not try change things part-way through. I mean, sure, it may be true that if you throw a spec over the wall, then you will get something thrown back that meets the spec, but the real world isn't so simple.

But are those sweeping generalizations true?

> I’m also going to be making some sweeping statements about “how software works”, these claims mostly hold, but they break down when applied to distributed systems, parallel code, or complex interactions between software systems and human processes.

I'd argue that this describes most software written since, uh, I hesitate to even commit to a decade here.

At least the 1950’s. That’s when stuff like asynchrony and interrupts were worked out. Dijkstra wrote at length about this in reference to writing code that could drive a teletype (which had fundamentally non-deterministic timings).

If you include analog computers, then there are some WWII targeting computers that definitely qualify (e.g., on aircraft carriers).

For the purposes of the article, which is to demonstrate how developing an LLM is completely different from developing traditional software, I'd say they are true enough. It's a CS 101 understanding of the software development lifecycle, which for non-technical readers is enough to get the point across. An accurate depiction of software development would only obscure the actual point for the lay reader.

Does that really matter?

He is trying to lax the general public perception around AIs shortcomings. He's giving AI a break, at the expense of regular developers.

This is wrong on two fronts:

First, because many people foresaw the AI shortcomings and warned about them. This "we can't fix a bug like in regular software" theatre hides the fact that we can design better benchmarks, or accountability frameworks. Again, lots of people foresaw this, and they were ignored.

Second, because it puts the strain on non-AI developers. It blamishes all the industry, putting together AI with non-AI in the same bucket, as if AI companies stumbled on this new thing and were not prepared for its problems, when the reality is that many people were anxious about the AI companies practices not being up to standard.

I think it's a disgraceful take, that only serves to sweep things under a carpet.

I don't think he's doing that at all. The article is pointing out to non-technical people how AI is different than traditional software. I'm not sure how you think it's giving AI a break, as it's pointing out that it is essentially impossible to reason about. And it's not at the expense of regular developers because it's showing how regular software development is different than this. It makes two buckets, and puts AI in one and non-AI in the other.

He is. Maybe he's just running with the pack, but that doesn't matter either.

The fact is, we kind of know how to prevent problems in AI systems:

- Good benchmarks. People said several times that LLMs display erratic behavior that could be prevented. Instead of adjusting the benchmarks (which would slow down development), they ignored the issues.

- Accountability frameworks. Who is responsible when an AI fails? How the company responsible for the model is going to make up for it? That was a demand from the very beginning. There are no such accountability systems in place. It's a clown fiesta.

- Slowing down. If you have a buggy product, you don't scale it. First, you try to understand the problem. This was the opposite of what happened, and at the time, they lied that scaling would solve the issues (when in fact many people knew for a fact that scaling wouldn't solve shit).

Yes, it's kind of different. But it's a different we already know. Stop pushing this idea that this stuff is completely new.

>But it's a different we already know

'we' is the operative word here. 'We', meaning technical people who have followed this stuff for years. The target audience of this article are not part of this 'we' and this stuff IS completely new _for them_. The target audience are people who, when confronted with a problem with an LLM, think it is perfectly reasonable to just tell someone to 'look at the code' and 'fix the bug'. You are not the target audience and you are arguing something entirely different.

Let's pretend I'm the audience, and imagine that in the past I said those things ("fix the bug" and "look at the code").

What should I say now? "AI works in mysterious ways"? Doesn't sound very useful.

Also, should I start parroting innacurate outdated generalizations about regular software?

The post doesn't teach anything useful for a beginner audience. It's bamboozling them. I am amazed that you used the audience perspective as a defense of some kind. It only made it worse.

Please, please, take a moment to digest my critique properly. Think about what you just said and what that implies. Re-read the thread if needed.

> He is trying to lax the general public perception around AIs shortcomings

This is not at all what I'm trying to do. This same essay is cross-posted on LessWrong[1] because I think ASI is the most dangerous problem of our time.

> This "we can't fix a bug like in regular software" theatre hides the fact that we can design better benchmarks, or accountability frameworks

I'm not sure how I can say "your intuitions are wrong and you should be careful" and have that be misintepreted as "ignore the problems around AI"

[1]: https://www.lesswrong.com/posts/ZFsMtjsa6GjeE22zX/why-your-b...

The word intuition here have power. Surely you must have noticed that vibe coding and intuition have a strong connection.

There are people (definitely not me) who buy 100% of anything AI they read. To that beginner enthusiastic audience, your text looks like "regular software is old and unintuitive, AI is better because you can just vibe", and the text reinforces that sentiment.

Bless you for having reading comprehension

I thought this blog post was a parody. And to my surprise both the author and the audience takes it seriously. Weird

Why did you think it a parody?

> Ok, I'm starting to feel this is a troll post. This guy can't be serious.

Did you read the footnote about writing regression tests to catch bugs before they come back in production?

Discussed here:

https://news.ycombinator.com/item?id=45583970

Thought I might just skip the repetition. You can continue the conversation within that thread.

Reply on news.ycombinator.com