Hacker News Reader

Comment by notashelf | Hacker News Reader

I'll risk sounding antagonistic and ask you this: if LLMs are not "good enough", why are they still around?

Obviously "good enough" was a poor pick of words of words on my behalf---and I'll gladly own it---but they should be "good enough" for something if a significant amount of resources keep getting allocated to them. Yes on a very personal level you look at LLM-generated code and think to yourself "wow this is garbage" but what about the people, as pointed out many times in this thread, that simply do not care? Does that not count as "good enough"?

View on HN

> Yes on a very personal level you look at LLM-generated code and think to yourself "wow this is garbage" but what about the people, as pointed out many times in this thread, that simply do not care? Does that not count as "good enough"?

Aside from the "amateur" qualifier, the Booch quote below provides insight to those questions:

  The amateur software engineer is always in search of magic, 
  some sensational method or tool whose application promises 
  to render software development trivial. It is the mark of 
  the professional software engineer to know that no such 
  panacea exist.[0]

"Code" is shorthand for "encode", which can be defined as "encoding a solution to a problem", which implies knowing what problem to solve, and thus ultimately implying understanding of same.

So "good enough" depends on the stakeholders' definition of that valuation.

IOW, if a person tasked with delivering a solution can instead deliver garbage and still get paid for it, then it is "good enough" for some definition of same. If the people approving a person's remuneration are not satisfied with this type of work product, then the answer as to whether it is "good enough" or not will be quickly answered.

0 - https://en.wikiquote.org/wiki/Grady_Booch#Quotes

The use of “amateur” is indeed a very poor choice. An amateur is someone who seeks to master a skill or perform a craft for the love of the art, not for money. It’s the professional who is looking for the shortcut, not the amateur.

The amateur is the one who hones a tool’s edge until it’s razor sharp, and cuts away all unnecessary detail until they’ve found the essence of what they’re trying to achieve. The work of an amateur is often highly impractical and uneconomical, but sublimely beautiful.

I think you are torturing this comparison by using different meanings of amateur and professional

A professional does mean "someone who does something for money" but it also often just means "someone who is very good at something"

Similarly, amateur does mean "someone who does something out of passion without wanting/needing to be paid" but also often just means "someone who is just starting out, is new at something, and is probably not good enough to be a (paid) professional doing it"

You're talking about the "passionate" amateur and comparing it to the "getting paid" professional. I think many people talking about amateurs and professionals are mostly talking about "amateurs (newbs)" and "professionals (very experienced)"

You just twisted the meanings of the words.

An amateur is not a master nor a disciple of a master (e.g. future master).

The vast majority of amateurs have no ambitions nor passion and simply give up early on, that's a rose tinted view.

Since they don't get paid to do it, they have to allocate their precious free time toward the skill, which means they don't have those 40 hour weeks where they spend all their time honing their skill to begin with.

Even someone who does a skill half heartedly without passion nor ambition is going to learn faster with those 40 hours at his disposal.

>The amateur is the one who hones a tool’s edge until it’s razor sharp, and cuts away all unnecessary detail until they’ve found the essence of what they’re trying to achieve.

That's not an amateur and you know that. By your logic Japanese master craftsmen doing this are amateurs.

It’s not I who have twisted the meanings of the words, it’s the people who use the word amateur when they mean hack. The word amateur is very old, and for most of its history [1] it literally meant lover and in the case of practitioners of some art or sport, it meant those who participate out of love for the activity, not for money.

[1] https://www.wordorigins.org/big-list-entries/amateur

The market logic isn’t going to be defensible. The market is highly irrational and prone to be influenced by those with the most capital, not the, “invisible hand.” Cars are freaking everywhere not because they are the best solution of transportation but because people with vested interest in the sale of oil and cars made it so by buying politicians, starting political and marketing campaigns, and on for generations. It took decades for the car to become the de-facto mode of transportation in North America let alone the rest of the world. It’s not good for us. It’s highly irrational to prefer cars when freaking wildfires and climate change threaten our very existence. In a similar vein, LLMs as they are, are not around because they are necessary and superior.

"The market chose it" doesn't mean it was the best solution, just that it fit the incentives

The current kind of car is extremely good at doing what it does. It's just that most of the benefits are short term, and for the car user (and the oil salesman), while the costs are borne by everyone else, and some of the worst ones are long term.

(Some similarities to hard drugs here.)

Cars are actually the best solution to transportation once you stop ignoring inconvenient but real requirements like autonomy and isolation.

> Cars are actually the best solution to transportation

"the" is the error here.

There's places and people for whom cars are so. There's other cases where public transport covers the people while business-owned vans cover deliveries of bulky items, which is essentially how I've lived the last 8 years, though for "bulky" I should note I've seen someone take an actual kitchen skink on public transport here in Berlin.

I've also found some islands here Berlin with no bridge connection to the mainland and hence no practical use for a car at those locations; that said, my main reference for "car free" domiciles would be Sark: https://en.wikipedia.org/wiki/Sark

Cars are not a system.

You need roads, traffic lights, drivers and pedestrians that respect rules, not too many holy cows in the way, not too much CO2 in your quasi-self-sustaining spaceship, lungs and livers that can deal with the soot, a tolerance for high-velocity impacts risks, not too many other cars around, ...

They're here, because even "bad" code has uses. We can have quantity over quality. We can generate throw-away programs (where the lack of maintainability doesn't matter). We can solve some problems by brute force (keep trying until tests pass). For people who can't program, even bad code can do more than no code.

LLMs are also useful as a natural language interface. They're pretty useful for search (as RAG and autoresearch, not the glue-on-pizza Google search crap). They're good for filtering and classification against rules that can be specified in natural language instead of special syntax or by training dedicated models.

> They're here, because even "bad" code has uses. We can have quantity over quality. We can generate throw-away programs (where the lack of maintainability doesn't matter). We can solve some problems by brute force (keep trying until tests pass). For people who can't program, even bad code can do more than no code.

If this is an acceptable definition of software engineering productivity, where "quantity over quality" is prized, then hire me.

Because I guarantee I can produce dozens of PRs daily, each having tens of thousands of LoC deltas (pick any language you desire), none of which having research and/or understanding underpinning them, and all easily quantified as being "brute force."

There's a quote I forgot that says: "There is a Quality to Quantity".

I think it was from a Chinese person, as they took manufacturing of cheap things, low quality stuff, but managed to scale it to mass quantities.

I agree with that, it may not be reliable, full featured, well designed, ergonomic, etc. But quantity enables things and thus is a quality in itself. It allows use-cases that benefit from accessible cheap software. Those use-cases were not viable prior, because cost/time was prohibitive.

Often attributed to Stalin in reference to the mass production focus of the Soviet war machine. https://quotees.co.uk/war/joseph-stalin-quantity-has-a-quali... When you treat scale as a design criteria, you get a different product

Ah interesting, it does say probably, so seems it's not fully confirmed. In any case, I guess my brain re-interpreted it in the context of manufacturing and making products and remembered it as that.

> There's a quote I forgot that says: "There is a Quality to Quantity".

> I agree with that, it may not be reliable, full featured, well designed, ergonomic, etc. But quantity enables things and thus is a quality in itself. It allows use-cases that benefit from accessible cheap software.

The underlying assumption here is that the quantity delivered is acceptable and does not worsen the customer experience. To continue with the manufacturing domain example; if a company produced widgets quickest and for minimal cost, yet had more lawsuits stemming from their use, is that the kind of enabling an organization should pursue?

> Those use-cases were not viable prior, because cost/time was prohibitive.

Sometimes a cigar is a cigar and sometimes what is thought to be a desirable use-case is prohibitive because, once sufficiently vetted, it is neither cost nor time which disqualifies it.

You're just saying quantity is not always a desirable quality. That's true, and hence it's not every use-case that would favor quantity over other qualities. But there also exists those which do, and trading other qualities for quantity can be a good trade off.

> the quantity delivered is acceptable and does not worsen the customer experience

No, it will explicitly worsen the customer experience in other ways, but it will also give them a benefit: access to something they couldn't afford at all before.

> Because I guarantee I can produce dozens of PRs daily, each having tens of thousands of LoC deltas (pick any language you desire), none of which having research and/or understanding underpinning them, and all easily quantified as being "brute force."

I can guarantee that you cannot produce such PRs in the same quantity/quality than a modern LLM. The speed at which today's LLMs output code is superhuman.

> I can guarantee that you cannot produce such PRs in the same quantity/quality than a modern LLM.

Who said I would not use a combination of LLMs, scripts, and any other automation technique available?

It's textbook historical revisionism. It's not like we didn't have mountains of bad code in the pre-LLM era. I'm old enough to remember endless complaint posts about having to wotk with legacy codebases full of horrible spaghetti code. Somebody took a long time to write it by hand.

Have you realized you are attacking a straw-man? Nobody said all “hand written” code is good. I _think_ the implicit argument is: “all vibed code lacks cohesion”.

I'm not happy about the drop in quality either, and I'm not advocating for just making software shittier.

However, look at history of DSLR cameras vs mobile phone cameras. Mobile phones were always worse than large format cameras, and we've got a massive quantity-over-quality explosion in photos taken. It didn't replace professional photography, but let people upload photos of their lunch to social media.

I think in programming we're getting frustrated because we're trying to naively use high-volume low-quality code generators in our previous low-volume high-quality workflows. We're still discovering what's the programming equivalent of bathroom selfies, photos of receipts and QR codes.

> I'll risk sounding antagonistic and ask you this: if LLMs are not "good enough", why are they still around?

To quote the parent commenter:

> They seem to be able to solve the immediate problem,

but not long term

But that means they're good enough for the short them. This is why I find it such an interesting question to wield, but also, it is why my wording was poor in the post.

Short term can, IME, be very short. I've seen people generate, say, a bash script with an LLM. It's generated: short term, the problem is "solved": we've generated a bash script.

… but does it work? Someone comes along, reviews it, "this is garbage, and does not do what it says it purports to do". Perhaps it even gave an output: the script computed … something, but it's just GIGO.

But that "check if this works" friction is the same friction that is what people try to avoid by generating it with an LLM in the first place. If you're too lazy to write the script, you're practically by definition too lazy to verify it.

You can have another agent write the tests and verify the former agent ? This is pretty basic stuff. Makes me question if people are actually trying to use AI

Now you have two problems lol, in that you don't know if the tests are any good or actually test the thing in question.

Sooner or later you run out of turtles to put on the stack.

That’s a solved problem, you just add another agent to check if the tests are any good, and one more to oversee the test-checker, and one more…

If this is your working environment, it sounds like quite an unusual place.

I literally can't imagine generating a script with an LLM without testing it at all.

Bash is one of those situations where LLMs can do really well. No human on Earth can remember all of the commands and even fewer humans can remember all of the switches for all of the commands.

This kind of remembering, searching, and assembling is exactly what LLMs are good at - as long as you're not writing a gigantic build system with hundreds of moving parts, in which case you should probably be using something more streamlined anyway.

> I literally can't imagine generating a script with an LLM without testing it at all.

Then you're extremely unimaginative as well as unusually fastidious.

Certainly someone - several someones - are generating lots of scripts and not testing them, given the PRs I'm seeing.

Breaking down long term problems into small shorter problems we can solve one after another until the long term problem is solved is the essence of engineering.

No, the "essence" of engineering is having a cohesive vision for a larger plan.

No quality large scale project can exist without it. And any seasoned engineer should understand by now that no SOTA LLM can produce quality engineering at scale.

Lol. That's not true at all. And not how projects are done on startups or big companies.

Execute fast or get nowhere, a series of small wins get you to live long term.

Long term projects that survived on short term wins are still around. But you're going to want to check how big the graveyard is of "project collapsed under tech debt from short term decisions" to see whether it's a good strategy.

Nah. You can succeed in every single individual thing but fail at the overall project because the parts don't align into anything that makes sense.

If you need examples, look at game dev. There's plenty of games that have good execution but aren't fun to play or they're a confusing mess because there wasn't good overall direction.

Videogames are a weird comparison because they are basically the ultimate culmination of the combination of both engineering and art. The two can crossover in weird ways but still have distinct aspects that do not really compromise the other.

Just because a videogame isn’t fun doesn’t mean there wasnt a cohesive engineering vision. God of War: Ragnarok is a good example of this. Excellent technical execution and (arguably) great direction but a terribly boring game.

And many games that are terribly engineered are also incredibly fun. Dark Souls is a good example - runs like ass, looks very rough even by the standards of the time when the game released, with terrible enemy AI in a combat focused game - one might even call it a confusing mess (and I laugh at the notion that it doesn’t have good direction)… despite all this it’s a classic and arguably the most influential game since its release.

Elden Ring carries a lot of legacy baggage from years of iteration on the same engine as ~~Dark~~ Demons Souls - and it’s widely considered From Software’s magnum opus and a masterpiece. Yet considering the scope of the game it is a great example of how small wins added up can solve large problems when executed with the skill and judgment gained from experience. So one might still consider it a success in terms of “engineering”.

Would From Software have been able to make the same game using Unreal Engine 5 built from scratch with dime a dozen “Unreal experts” brought in off the street? Not a chance. Want to know why western game development studios are suffering right now? Compare American layoff culture and race to the bottom economics to the retention rates of Japanese game studios like Capcom, Nintendo and other heavy hitters.

For many of the most marvellous things humans have ever built, if you take a peak behind the curtain you’ll still probably find some amount of duct tape and popsicles sticks somewhere keeping it all together.

Fun, however, is a matter of taste and you can’t engineer that.

They're "good enough", but they're not good. If you're trying to run a business, programmer skill isn't worth the money you spend, LLMs are cheaper. If you care about the quality of what you produce, they're trash.

Programming is no longer skilled labor. Quality is a hobby.

If your business involves storing any kind of customer data or handling payments, you probably should still hire a programmer who at least reviews the results of your vibe coding. Getting hacked is bad for business.

AI is just fine at security audits these days. It's better at finding bugs than it is at writing code, overall. You need to prompt for it, but that's not skilled labor.

I'm also not sure getting hacked is so bad for business. Most businesses that got hacked seem to still be around just fine.

Because if you criticize the LLMs, you get headbutted into the Dante's Inferno that is the current job market

I'd agree with the OP you are responding, what he means is that this isn't just about taste. It's that the code, while it works, and the text, while it has the ideas in it, is still not to the level of a human author. It's not that it isn't tasteful, in the sense of having the flair of specific author X, but it is verbose, goes around the point to get to it, etc.

Taste seems to imply an art, but I don't think we're at the point of cleverness, of LLM innovating a new beautiful programming paradigm or pattern out of immaculate taste for code. We're just talking more basic, even bad devs have more terseness, more clear intent in their prose and code, etc.

It's "good enough" for many use-cases, but it could be better, even before expecting the most beautifully clear code from the best devs out there, it has not reached the level of the average one in readability and clarity of expression (both code and prose). That annoys the author, he has to intervene manually to trim all the fat, and therefore not "good enough".

It’s a good question and I don’t doubt they’re useful.

But I mean they’re not good enough to do the things I need them to. Not enough that any difference can be chalked up to “taste”.

Put it this way, if you hired a very good handyman to build you a cathedral, you wouldn’t stand looking at the smouldering wreckage of the construction site saying “oh well that’s just a matter of taste”.

The market desperately wants to replace wages and the idea of a machine doing the work you previously needed to do before is so enticing, and the initial results so promising, that the rest doesn't matter.

Look at how many AI lay offs there already were a year or two ago - back when almost everyone would agree they weren't good enough. If good enough didn't matter then does it matter how when AI is better?

And despite what you might be trying to prove, it's obvious I'm replying to an LLM (a green account at that), but it's not for you, I don't expect you to bother reading it.

> if LLMs are not "good enough", why are they still around

They are not good enough yet, the people using these tools every day, and with a high bar for quality and design 'taste' notice the rough edges.

They are still around because LLMs are plenty good enough for casual consumers, but where 'taste' really starts to matter in design or in writing, the rough edges show up much more obviously and users who have been working with these tools feel it immediately.

One part of the answer is marketing it to people looking for an easy way out without having to do the work or thinking at all. There's value in that as shown in everything like things for the home that purport to make your daily life easier. LLMs market themselves in much the same way

Conversely, if they are good enough, why do they degrade frome consuming their own output?

How come humans consuming their own output gets better over time, but llms and any other form of ai consuming their own output only get worse?

Not disagreeing with you but the "resource allocation" which I presume to hint at the fallacy that capitalism is somehow great or optiomal at allocating resources is not really true. Just look at any big company such as meta and see how much inefficiency and bad choices there is.

Stock market as a measuring stick is completely irrational.

Facebook/Meta is a great example indeed when one remembers that they took part in a genocide, and still haven't faced consequences for it.

There's the hope that they'll make all employees redundant and that businesses that adopted AI quickly will be able to exist as a pure money-printing entity without the liabilities of products, customers, or employees.

Put another way, LLMs are just another part of enshittification.

Reply on news.ycombinator.com