Hacker News Reader

Comment by slopinthebag | Hacker News Reader

> I didn’t write any piece of code there. There are several known issues, which I will task the agent to resolve, eventually. Meanwhile, I strongly advise against using it for anything beyond a studying exercise.

Months of effort and three separate tries to get something kind of working but which is buggy and untested and not recommended for anyone to use, but unfortunately some folks will just read the headline and proclaim that AI has solved programming. "Ubiquitous hardware support in every OS is going to be a solved problem"! Or my favourite: instead of software we will just have the LLM output bespoke code for every single computer interaction.

Actually a great article and well worth reading, just ignore the comments because it's clear a lot of people have just read the headline and are reading their own opinions into it.

View on HN

The author specifically said that they did not read the code or even test the output very thoroughly. It was intentionally just a naive toy they wanted to play around with.

Nothing to do with AI, or even the capabilities of AI. The person intentionally didn't put in much effort.

> Nothing to do with AI, or even the capabilities of AI. The person intentionally didn't put in much effort.

The part to do with AI is that it was not able to drive a comprehensive and bug free driver with minimal effort from the human.

That is the point.

Why is that the metric? In my job, I get drafts from junior employees that requires major revisions, often rewriting significant parts. It’s still faster to have someone take the first pass. Why can’t AI coding be used the same way? Especially if AIs are capable of following your own style and design choices, as well as testing code against a test suite, why isn’t it easier to start from a kind of working baseline than to rebuild from a raf h.

Dis you hire juniors just to get drafts? That seems pretty inneficient.

I'm a lawyer, so a bunch of work--factual analysis, legal research, etc.--goes into the draft that isn't just the words on the page. At the same time, the work product is meant to persuade human readers, so a lot of work goes into making the words on the page perfect. (Perhaps past the point of diminishing returns, but companies are willing to pay for that incremental edge when the stakes are high.)

Programming is different in that you don't usually have senior engineers rewrite code written by junior engineers. On the other hand, look at how the Linux kernel is developed. You have Linus at the top, then subsystem maintainers vetting patches. The companies submitting patches presumably have layers of reviewers as well. Why couldn't you automate the lower layers of that process? Instead of having 5 junior people, maybe you have 2 somewhat more senior people leveraging AI.

This is probably not sustainable unless the AI can eventually do the work the more senior people are doing. But that probably doesn't matter in the short term for the market.

Maybe because code is different. A software is a recipe that an autonomous machine can follow (very fast and repeateadly).

But the whole goal of software engineering is not about getting the recipe to the machine. That’s quite easy. It’s about writing the correct recipe so that the output is what’s expected. It’s also about communicating the recipe to fellow developers (sharing knowledge).

But we are not developing recipe that much today. Instead we’ve built enough abstractions that we’re developing recipes of recipes. There’s a lot of indirection between what our recipe says and the final product. While we can be more creative, the failure mode has also increased. But the cost of physically writing a recipe has gone down a lot.

So what matters today is having a good understanding of the tower of abstractions, at least the part that is useful for a project. But you have to be on hand with it to discern the links between each layer and each concept. Because each little one matters. Or you delegate and choose to trust someone else.

Trusting AI is trusting that it can maintain such consistent models so that it produces the expected output. And we all know that they don’t.

I’m not able to provide a comprehensive bug free driver.

Seems like they did put in quite a bit of effort, but were not knowledgeable enough on wifi drivers to go further.

So hardware drivers are not a solved problem where you can just ask chatgpt for a driver and it spits one out for you.

If you could write drivers in javascript, it probably would have done just fine /s

> The person intentionally didn't put in much effort.

Aren't you just describing every vibe code ever?

To think about it, that is probably my main issue with AI art/books etc. They never put in any effort. In fact, even the competition is about putting least effort.

> The author specifically said that they did not read the code or even test the output very thoroughly. It was intentionally just a naive toy they wanted to play around with.

Yes and that's what I'm pointing out, they vibe coded it and the headline is somewhat misleading, although it's not the authors fault if you don't go read the article before commenting.

But it does have to do with AI (obviously), and specifically the capabilities of AI. If you need to be knowledgable about how wifi drivers work and put in effort to get a decent result, that obviously speaks volumes about the capabilities of the vibe coding approach.

I strongly suspect that somebody with domain knowledge around Wi-Fi drivers and OS kernel drivers could prompt the llm to spit out a lot more robust code than this guy was able to. That's not a knock on him, he was just trying to see what he could do. It's impressive what he actually accomplished given how little effort he put forth and how little knowledge he had about the subject.

Someone with domain knowledge could also just write the code instead of trying to get the stochastic prediction machine to generate it. I thought the whole point was to allow people without said expertise to generate it. After all, that seems to be the promise.

> Someone with domain knowledge could also just write the code instead of trying to get the stochastic prediction machine to generate it.

Well, people with the domain knowledge exist, yet they have not yet written this driver... why not?

Because there is other code those experts want to write, and they don't have time to write it all... but what if they could just give a fairly straightforward prompt and have the LLM do it for them? And if it only took minor tweaks to the prompt to have it write drivers for all the myriad combinations of hardware and software? At that point, there might be enough time to write it all.

Just because people exist that can DO all the work doesn't mean we have enough person-hours to do ALL the work.

> Because there is other code those experts want to write, and they don't have time to write it all... but what if they could just give a fairly straightforward prompt and have the LLM do it for them?

Then pretty soon they wouldn't be the experts anymore?

Maybe? But you could make the same argument that programmers today aren't "experts" at computers because they don't know how to build CPUs.

There is no reason to believe you can't gain expertise while still using higher and higher level abstractions. Yes, you will lose some of that low level expertise, but you can still be an expert at the problem set itself.

The way I see it, this is only possible because you can trust the lower layers work reliably and predictably enough that you can move up.

If your operating system was regenerated every day slightly differently and with certain things working and others not, you’d quickly revert to the lower predictable abstraction.

Clearly there wasn't much appetite for someone to do that.

It will be like that at some point soon, just not now. Are you trying to make the point that because this technology is not yet perfect the fact that it can already do so much is unimpressive?

Will it happen before or after we get fusion energy? I heard that was coming soon too.

@petcat Is your nickname a description or an instruction?

You're validly critiquing where it is now.

The hype people are excited because they're guessing where it's going.

This is notable because it's a milestone that was not previously possible: a driver that works, from someone who spent ~zero effort learning the hardware or driver programming themselves.

It's not production ready, but neither is the first working version of anything. Do you see any reason that progress will stop abruptly here?

Not a huge fan of @sama, but he is quoted as saying: this is the worst these models will every be!

Puts all criticism in a new perspective.

That's like Bill Gates saying XP is the worst Windows will ever be

Not Windows: Operating systems. We did get more capable operating systems. The point of the quote is "this is the worst the SOTA will ever be".

If Windows XP were fully supported today I still wouldn't use it, personally, despite having respect for it in its era. The core technology of how, eg OS sandboxing, security, memory, driver etc stacks are implemented have vastly improved in newer OSes.

You're just moving the goal posts unfortunately. The point is that positive progress is never actually guaranteed.

Of course not. But I believe your Windows example was implying fundamental tech got worse.

The original "worst" quote is implying SOTA either stays the same (we keep using the same model) or gets better.

People have been predicting that progress will halt for many years now, just like the many years of Moore's law. By all indications AI labs are not running short of ideas yet (even judging purely by externally-visible papers being published and model releases this week).

We're not even throwing all of what is possible on current hardware technology at the issue (see the recent demonstration chips fabbed specifically for LLMs, rather than general purpose, doing 14k tokens/s). It's true that we may hit a fundamental limit with current architectures, but there's no indication that current architectures are at a limit yet.

Aged like milk.

That assumes he is all knowing.

>> Do you see any reason that progress will stop abruptly here?

I do. When someone thinks they are building next generation super software for 20$ a month using AI, they conveniently forget someone else is paying the remaining 19,980$ for them for compute power and electricity.

People abstract upon new leaps in invention way too early though. Believing these leaps are becoming the standard. Look at cars, airplanes, phones, etc.

After we landed on the moon people were hyped for casual space living within 50 years.

The reality is it often takes much much longer as invention isn't isolated to itself. It requires integration into the real world and all the complexities it meets.

Even moreso, we may have ai models that can do anything perfectly but it will require so much compute that only the richest of the rich are able to use it and it effectively won't exist for most people.

> Do you see any reason progress will stop abruptly here?

Yeah, money and energy. And fundamental limitations of LLM's. I mean, I'm obviously guessing as well because I'm not an expert, but it's a view shared by some of the biggest experts in the field ¯\_(ツ)_/¯

I just don't really buy the idea that we're going to have near-infinite linear or exponential progress until we reach AGI. Reality rarely works like that.

So far the people who bet against scaling laws have all lost money. That does not mean that their luck won’t change, but we should at least admit the winning streak.

You mean Moore's law? Which is now dead?

No I don't mean that. I mean the LLM parameter scaling laws. More importantly, it doesn't matter if I mean that or Moore's law or anything else, because I'm not making a forward looking prediction.

Read what I wrote.

I'm saying is if you bet AGAINST [LLM] scaling laws--meaning you bet that the output would peter out naturally somehow--you've lost 100% so far.

100%

Tomorrow could be your lucky day.

Or not.

This weekend I had 100% success at the blackjack table, until I didn't and lost.

I guess we'll see :)

You gonna go read up on some 0% success rate strategies on the way?

What I’m saying is that we act as though claims about these scaling laws have never been tested. People feel free to just assert that any minute now the train will stop. They have been saying that since the Stochastic parrots.

It has not come true yet.

Tomorrow could be it. Maybe the day after. But it would then be the first victory.

it's not dead. it's enough to look at GB200/GB300 vs Vera Rubin specs.

At the very least, computers are still getting faster. Models will get faster and cheaper to run over time, allowing them more time to "think", and we know that helps. Might be slow progress, but it seems inevitable.

I do agree that exponential progress to AGI is speculation.

You think all AI companies will never release a better model days after they all release better models?

That is a position to take.

I know some proponents have AGI as their target, but to me it seems to be unrelated to the steadily increasing effectiveness of using LLMs to write computer code.

I think of it as just another leap in human-computer interface for programming, and a welcome one at that.

If you imagine it just keeps improving, the end point would be some sort of AGI though. Logically, once you have something better at making software than humans, you can ask it to make a better AI than we were able to make.

I don’t think that follows, nor do I think it will keep improving indefinitely. It will certainly continue to improve for a while.

We don’t need anything close to AGI to render the job “software engineer” as we know it today completely obsolete. Ever hear of a lorimer?

If it doesn't follow, why not?

The other possibility is, as you say, progress slows down before its better than humans. But then how is it replacing them? How does a worse horse replace horses?

I said I don’t think it follows, and you certainly gave no support for the idea that it must follow. Logically speaking, it’s possible for improvements to continue indefinitely in specific domains, and never come close to AGI.

Progress in LLMs will not slow down before they are better at programming than humans. Not “better than humans.” Better at programming. Just like computers are better than humans at a whole bunch of other things.

Computers have gotten steadily better at adding and multiplying and yet there is no AGI or expectation thereof as a result.

Either the AI can do better than humans at programming, or it can't. If I ask it to make an improved AI, or better tools for making an improved AI, and it can't do it, then at best it's matching human output.

All the current AI success is due to computers getting better at adding and multiplying. That's genuinely the core of how they work. The people who believe AGI is imminent believe the opposite of that last claim.

No one is talking about AGI in this thread except you, though. The post said nothing about it. It's an absolute non sequitur that you brought up yourself.

I don’t get this response. This is amazing! What percentage of programmers can even write a buggy FreeBSD kernel driver? If you were tasked at developing this yourself, wouldn’t it be a huge help to have something that already kind of works to get things started?

Fairly high could - but some could start today some need a few months of study before they know how to start (and then take 10x long than the first person to get it working)

> instead of software we will just have the LLM output bespoke code for every single computer interaction.

That's sort of the idea behind GPU upscaling: You increase gaming performance and visual sharpness by rendering games at lower resolutions and use algorithms to upscale to the monitor's native resolution. Somehow cheaper than actually rendering at high resolution: Let the GPU hallucinate the difference at a lower cost.

Programmers have always been in search of an additional layer of abstraction. LLM coding feeds exactly into this impulse.

Reply on news.ycombinator.com