Hacker News Reader

Comment by simpaticoder | Hacker News Reader

A bit of a "dog bites man" story to note that a CEO of a hot company is hyping the future beyond reason. The real story of LLMs is revealed when you posit a magical technology that can print any car part for free.

How would the car industry change if someone made a 3D printer that could make any part, including custom parts, with just electricity and air? It is a sea change to manufacturers and distributors, but there would still be a need for mechanics and engineers to specify the correct parts, in the correct order, and use the parts to good purpose.

It is easy to imagine that the inventor of such a technology would probably start talking about printing entire cars - and if you don't think about it, it makes sense. But if you think about it, there are problems. Making the component of a solution is quite different than composing a solution. LLMs exist in the same conditions. Being able to generate code/text/images is of no use to someone who doesn't know what to do with it. I also think this limitation is a practical, tacit solution to the alignment problem.

View on HN

This argument simply asserts that the LLMs (or their successor systems including scaffolding) will asymptote somewhere below human capabilities.

It’s possible that this could happen but you need to propose a mechanism and metric for this argument to be taken seriously (and to avoid fooling yourself with moving goalposts). Under what grounds do you assert that the trend line will stop where you claim it will stop?

Yes, if super-human AGI simply never happens then the alignment problem is mostly solved. Seems like wishful thinking to me.

This. It’s far harder to think of reasons that limits will endure, in a world where innovation by inches has always produced improvements. Everyone brings out the current state and technology. Those are transient.

WTF? Look at a word processor today, and then one from 1999, and tell me again how technology always keeps improving dramatically.

The standard electrical wall sockets that you use have not really changed since WW2. For load bearing elements in buildings, we don't have anything substantially better today than 100 years ago. There is a huge list of technological items where we've polished out almost every last wrinkle and a 1% gain once a decade is hailed as miraculous.

So what? Many animals haven’t changed much for millions of years, and that was irrelevant to our emergence. Not everything has to change in equal measure.

There are many reasons for all those things not to change. Limits abound. We discovered that getting taller or faster isn’t “better”, all we needed is smarter. Intelligence is different. It applies to everything else. You can lose a limb or eyesight and still be incredibly capable. The intelligence is what makes us able to handle all the other limits and change the world even though MS Word hasn’t changed much.

We are now applying a lot of our intelligence to inventing another one. The architecture won’t stay the same, the limits won’t endure. People keep trying and it’s infinitely harder to imagine reasons why progress will stop. Just choose any limit and defend it.

There is an interesting delusion among web company workers that goes something like “technology progress goes steadily up and to the right” which I think comes from web company managers who constantly prattle on about “innovation”. Your word processor example is a good one, because at some point making changes to the UI just hurts users. So all that empty “innovation” that everyone needs to look busy doing is actually worse than doing nothing at all. All that is a roundabout way to say I think tech workers have some deep need to see their work as somehow contributing to “progress” and “innovation” instead of just being the meaningless undirected spasms of corporate amoebas.

Do you have any points that aren’t about the people you disagree with? Argue the facts. What are the limits that prevent progress on the dimension of replicating human intelligence?

> Do you have any points that aren’t about the people you disagree with?

Yes...

> Argue the facts.

What?

> What are the limits that prevent progress on the dimension of replicating human intelligence?

I don't work in that field, but as a layman I'd wager the lack of clear technical understanding of what animal intelligence actually is, let alone how it works is the biggest limitation.

> What

Half of the language in your comment was fact free and incendiary. Delusion, prattle, deep needs of some role. That’s all firmly in the realm of ad hominem, so I was asking if you had anything substantial.

> I'd wager the lack of clear technical understanding of what animal intelligence actually is

I made the same points long time ago so can’t be too critical of that. I’ve changed my mind and here’s why. That’s not any kind of universal limit. It’s a state, and we can change states. We currently don’t understand intelligence but there’s no barrier I’m aware of that prevents progress. In addition, we’ve discovered types of intelligence (LLM’s, AlphaGo Zero etc) that don’t depend on our ability to understand ourselves. So our inability to understand intelligence isn’t a limit that prevents progress. New algorithms and architectures will be tested, in an ongoing fashion, because the perceived advantages and benefits are so great. It’s not like the universe had a map for intelligence before we arrived, it emerged from randomness.

I’m less sure that it’s a good idea, but that’s a different discussion. Put me in the camp of “this is going to be absolutely massive and I wish people took it more seriously.”

> fact free and incendiary

No. I believe it is factually accurate that many tech workers believe that technology progress marches steadily onwards and upwards. This is easily shown by the historical record to be patently false. Moreover, it's easy to imagine myriad ways we could regress technologically--nuclear war, asteroid impact, space weather, etc. So a belief in the inexorable progress of technology is therefore delusional. I believe it is a fact that tech companies' managers encourage these delusions by spinning up a bunch of pseudo religious sentiment using language like "innovation" to describe what is mostly actually really mundane, meaningless, and often actively harmful work. People hear that stuff, it goes to their heads, and they think they're "making the world a better place". The inexorable march of technology progress fits into such a world view.

> We currently don’t understand intelligence but there’s no barrier I’m aware of that prevents progress. In addition, we’ve discovered types of intelligence (LLM’s, AlphaGo Zero etc) that don’t depend on our ability to understand ourselves.

How can you claim both that we don't know what intelligence is, and that LLMs, AlphaGo Zero, and etc are "intelligent"?

If you’re reaching for nukes and asteroids then I think you just want to argue. Have a nice day.

> For load bearing elements in buildings, we don't have anything substantially better today than 100 years ago

What about post-tensioned concrete slabs? Cross-laminated timber? Structural glazing? Intumescent paint?

Technology does keep improving, actually.

Except I can do things today that were not possible in 1999 like easily collaborate on a google doc realtime with someone on the other side of the planet, get an automated grammar expert to suggest edits and then have a video call with 15 people to discuss more about it. All in the same app, the browser, all on a whim, all high speed and to boot: all for free as in beer. And this is run of the mill YaaaaawN productivity tools.

I can also create a web scale app in a weekend using AWS. It is just insane what we can do now vs. 1999. I remember in early 2000s Microsoft boasting how it could host a site for the olympics using active server pages. This was PR worthy. That would be a side project for most of us now using our pocket money.

That's interesting, personally I've never found collaborative editing to be of any use, but the comments feature is instead the one I use, and often it works less well than the way we used to do it with inline comments (e.g. italicized or made a different color). The one advantage would be automatically merging comments from multiple reviewers, but that's not necessarily a good thing. Often the collection of comments from different reviewers each forms their own narrative, and merging it all into a chaotic mess drowns out each individual's perspective. Personally I'd rather treat each reviewer's critique individually. 1999 technology handles this just fine.

Same for video calls. Screen sharing can be useful at times, but it would be easy distribute materials to all participants and collaborate on a conference call. You'll have better audio latency that way. So it's not super obvious to me that the latest Wunderwaffe are actually significantly better, but it is clear they use a hell of a lot more compute cycles and bandwidth.

That's a really good comment and insight, but understandably I think it is aimed at this forum and a technical audience. It landed well for me in terms of the near team impact of LLMs and other models. But outside this forum, I think our field is in a crisis from being very substantially oversold and undersold at the same time.

We have a very limited ability to define human intelligence, so it is almost impossible to know how near or far we are from simulating it. Everyone here knows how much a challenge it is to match average human cognitive abilities in some areas, and human brains run at 20 watts. There are people in power that may take technologists and technology executives at their word and move very large amounts of capital on promises that cannot be fulfilled. There was already an AI Winter 50 years ago, and there are extremely unethical figures in technology right now who can ruin the reputation of our field for a generation.

On the other hand, we have very large numbers of people around the world on the wrong end of a large and increasing wealth gap. Many of those people are just hanging on doing jobs that are actually threatened by AI. They know this, they fear this, and of course they will fight for their and their families lifestyles. This is a setup for large scale violence and instability. If there isn't a policy plan right now, AI will be suffering populist blowback.

Aside from those things, it looks like Sam has lost it. The recent stories about the TSMC meeting, https://news.ycombinator.com/item?id=41668824, was a huge problem. Asking for $7T shows a staggering lack of grounding in reality and how people, businesses, and supply chains work. I wasn't in the room and I don't know if he really sounded like a "podcasting bro", but to make an ask of companies like that with their own capital is insulting to them. There are potential dangers of applying this technology; there are dangers of overpromising the benefits; and neither of them are well served when relatively important people in related industries thing there is a credibility problem in AI.

LLMs aren't really simulating intelligence so much as echoing intelligence.

The problem is when the hype machine causes the echoes to replace the original intelligence that spawned the echoes, and eventually those echoes fade into background noise and we have to rebuild the original human intelligence again.

I appreciate this, that is why I said, "LLMs and other models". Knowing the probability relations between words, tokens, or concepts/though vectors is important, and can be supplemented by smaller embedded special purpose models/inference engines and domain knowledge in those areas.

As I said, it is overhyped in some areas and underhyped in others.

Yes, basically this phenomenon: https://www.nature.com/articles/s41586-024-07566-y

Not enough plastics, glass, or metal in the air to make it happen. You need a scrapyard. Actually, that's how the LLMs treat knowledge. They run around like Wall-E grabbing bits at random and assembling them in a haphazard way to quickly give you something that looks like the thing you asked for.

> How would the car industry change if someone made a 3D printer that could make any part, including custom parts, with just electricity and air?

The invention would never see the light of day. If someone were to invent Star Trek replicators, they'd be buried along with their invention. Best Case it would be quickly captured by the ownership class and only be allowed to be used by officially blessed manufacturing companies, and not any individuals. They will have learned their lesson from AI and what it does to scarcity. Western [correction: all of] society is hopelessly locked into and dependent on manufacturing scarcity, and the idea that people have to pay for things. The wealthy and powerful will never allow free abundance of physical goods in the hands of the little people.

I'm curious if this is true.

Are there specific historical examples of this that come to mind?

The closest real-life thing (and probably what a lot of believers in this particular conspiracy theory are drawing on directly) is probably the Phoebus lightbulb cartel: https://en.wikipedia.org/wiki/Phoebus_cartel

It’s rather hard to imagine even something like that (and it’s pretty limited in scope compared to the grand conspiracy above) working today, though; the EC would definitely stomp on it, and even the sleepy FTC would probably bestir itself for something so blatant.

It only worked at the time because it benefited the public at large.

And that’s not really the same thing. Did someone invent the LED bulb in 1920 and that cartel crushed it? Not really.

In reality, the biggest problem was they had no incentive to invest in new lighting technology research, although they had the money to do so. It takes a lot of effort to develop a new technology, and significantly more to make it practical and affordable.

I think the story of the development of the blue LED which led to modern LED lighting is more illustrative of the real obstacles of technological development.

Companies/managers don't want to invest in R&D bc it’s too uncertain and they typically are more interested in the short term.

And it’s hard for someone without deep technical knowledge to identify a realistic worthwhile technical idea from a bad one. So they focus on what they can understand and what they can quantify ().

And even technical people can fail to properly evaluate ideas that are even slightly outside their area of expertise (or even sometimes the ones that are within it )

There are plenty of examples of good things being unavailable due to market manipulation and corrupt government.

But I don't know of anything nearly as extreme as destroying an entire invention. Those tend to stick around.

You're asking for historical examples of great inventions that were hidden from history. I only know of those great inventions that I've hidden...

There are a number of counterexamples though. Henry Ford, etc.

Just wondering if you’ve actually hidden inventions you would actually consider to be great?

>The wealthy and powerful will never allow free abundance of physical goods in the hands of the little people.

Then there would be a violent revolution which wrestles it out of their hands. The benefits of such a technology would be immediately obvious to the layman and he would not allow it to be hoarded by a select few.

Only if someone can make money from it. If the people who have it can't do that (or won't) then it won't happen. E.g. there's no private industry for nuclear bombs selling to any bidder in the world. But anything else you just need to let people make companies and that'll get it out there, in a thousand shades of value to match all the possible demands.

I don't know. Today it's extremely expensive to start new manufacturing competitors. You have to go and ask for tons of capital merely to play the game and likely lose. Anyone with that much capital is probably going to be skeptical of some upstart and consult with established industry leaders. This would be the opportunity for those leaders to step in and take the tech for themselves.

So to solve this problem you need billions to burn on gambles. I guess that's how we ended up with VC's.

> Western society is hopelessly locked into and dependent on manufacturing scarcity, and the idea that people have to pay for things.

How do you reconcile that with the fact that Western society has invented, improved, and supplied many of the things we lament that other countries don't have (and those countries also lament it - it's not just our own Stockholm Syndrome).

Most of what gets invented either 1. relies on its scarcity to extract value or 2. if it's not naturally scarce, it gets locked behind patents, copyrights, cartels, regulations, and license agreements between corporations, in order to keep the means of its production out of the hands of common people. Without that scarcity and ability to profit from it, it wouldn't be invented.

I dunno, magically fabricating a part is fundamentally different than magically deciding where and when to do so.

AI can magically decide where to put small pieces of code. Its not a leap to imagine that it will later be good at knowing where to put large pieces of code.

I don't think it'll get there any time soon, but the boundary is less crisp than your metaphor makes it.

> I dunno, magically fabricating a part is fundamentally different than magically deciding where and when to do so.

Right.

It sounds to me like you agree and are repeating the comment but are framing as disagreeable.

I'm sure I'm missing something.

The person you replied to suggests that the analogy of a 3d printer building a part does not hold, as LLM-based coding systems are able to both "print" some code, and decide where and when to do so.

I tend to agree with them. What people seem to miss about LLM coding systems, IMO:

a) deciding on the capabilities of an LLM to code after a brief browser session with 4o/claude is comparable to waking up a coder in the middle of the night, and having them recite the perfect code right then and there. So a lot of people interact with it that way, decide it's meh, and write it off.

b) most people haven't tinkered with with systems that incorporate more of the tools human developers use day to day. They'd be surprised of what even small, local models can do.

c) LLMs seem perfectly capable to always add another layer of abstraction on top of whatever "thing" they get good at. Good at summaries? Cool, now abstract that for memory. Good at q/a? Cool, now abstract that over document parsing for search. Good at coding? Cool, now abstract that over software architecture.

d) Most people haven't seen any RL-based coding systems yet. That's fun.

----

Now, of course the article is perfectly reasonable, and we shouldn't take what any CEO says at face value. But I think the pessimism, especially in coding, is also misplaced, and will ultimately be proven wrong.

(D) includes me, and does sound fun. Got any references?

> Got any references?

This is a good question, and I worry you won't get a response. Here is a pattern I've observed very frequently in the LLM space, with much more frequency than random chance would suggest:

  Bob: "Oh, of course it didn't work for you, you just need to use an ANA (amazing new acronym) model"
  Alice: "Oh, that's great, where can I see how ANA works? How do I use it?"
  ** Bob has left the chat **

I'm sorry but if you're calling RL "ANA (amazing new acronym)" you're out of your depth on this one.

Saw a private demo, felt as giddy as I felt when deepmind showed Mario footage. No idea when it'll be out.

Right, AI is good and will be good. I agree with you 110%. I quit a well-paying job and have gone a year without salary on this premise :)

However, I worry about the premise underlying your reply: a sense that this is somehow incompatible with the viewpoint being discussed.

i.e. it's perfectly cromulent to both think LLMs are and will continue to be awesome at coding, even believe they get much better. And also that you could give me ASI today, and there'd be an incredible long tail of work and processes to reformulate to pull off replacing most labor. It's like having infinite PhDs available by text message. Unintuitively, not that much help. I can't believe I'm writing that lol. But here we are.

Steve Sinfosky had a good couple long posts re: this on X that discuss this far better than I can.

Agree. The way I frame it is the world changes very slowly so even very capable new technologies take time to integrate into it.

> AI can magically decide where to put small pieces of code.

Magically, but not particularly correctly.

I would say LLMs are much less “produce a perfect part from nothing” and more “cut down a tree and get a the part you ask for, for a random model of car”

Car analogies are a little fraught because, I mean, in some cases you can even die if you put just the wrong parts in your car, and they are pretty complex mechanically.

If you had a printer that could print semi-random mechanical parts, using it to make a car would be obviously dumb, right? Maybe you would use it to make, like, a roller blade wheel, or some other simple component that can be easily checked.

> specify the correct parts, in the correct order, and use the parts to good purpose

While the attention-based mechanisms of the current generation of LLMs still have a long way to go (and may not be the correct architecture) to achieve requisite levels of spatial reasoning (and of "practical" experience with how different shapes are used in reality) to actually, say, design a motor vehicle from first principles... that future is far more tangible than ever, with more access to synthetic data and optimized compute than ever before.

What's unclear is whether OpenAI will be able to recruit and retain the talent necessary to be the ones to get there; even if it is able to raise an order of magnitude more than competitors, that's no guarantee of success. My guess would be that some of the decisions that have led to the loss of much senior talent will slow their progress in the long run. Time will tell!

> The real story of LLMs is revealed when you posit a magical technology that can print any car part for free.

I think the insight is that some people truly believe that LLMs would be exactly as groundbreaking as a magical 3D printer that prints out any part for free.

And they're pumping AI madly because of this belief.

Printed this out and pasted it into my journal. Going to come back to it in a few years. This touches on something important I can’t quite put into words yet. Some fundamental piece of consciousness that is hard to replicate - desire maybe

Desire is a big part of it! Right now LLMs just respond. What if you prompted an LLM, “your goal is to gain access to this person’s bank account. Here is their phone number. Message with them until you can confirm successful access.”

Learning how to get what you want is a fundamental skill you start learning from infancy.

Your post is a good summarization of what I believe as well.

But what's interesting when I speak to laymen is that the hype in the general public seems specifically centered on the composite solution that is ChatGPT. That's what they consider 'AI'. That specific conversational format in a web browser, as a complete product. That is the manifestation of AI they believe everyone thinks could become dangerous.

They don't consider the LLM API's as components of a series of new products, because they don't understand the architecture and business models of these things. They just think of ChatGPT and UI prompts (or it's competitor's versions of the same).

I think people think* of ChatGPT not as the web UI, but as some mysterious, possibly thinking, machine which sits behind the web UI. That is, it is clear that there’s “something” behind the curtain, and there’s some concern maybe that it could get out, but there isn’t really clarity on where the thing stops and the curtain begins, or anything like that. This is more magical, but probably less wrong, than just thinking of it as the prompt UI.

*(which is always a risky way of looking at it, because who the hell am I? Neither somebody in the AI field, nor completely naive toward programming, so I might be in some weird knows-enough-to-be-dangerous-not-enough-to-be-useful valley of misunderstanding. I think this describes a lot of us here, fwiw)

An LLM is not going to go skynet. It's not smart enough and can't take initiative.

An AGI however could. Once it reaches IQs of more than say 500 it would become very hard to control it.

Power cord?

Here are two explanations of why cutting the power won't help you once you reach that state. In short, you have already lost.

[0]: No Physical Substrate, No Problem https://slatestarcodex.com/2015/04/07/no-physical-substrate-...

[1]: It Looks Like You’re Trying To Take Over The World https://gwern.net/fiction/clippy

It is possible that some hypothetical super intelligent future AI will do something very clever like hide in a coding assistant and then distribute bits of itself around the planet in the source code of all of our stoplights and medical equipment.

However I think it’s more likely that it will LARP as “I’m an emotionally supportive beautiful AI lady, please download me to your phone, don’t take out the battery or I die!”

> It is possible that some hypothetical super intelligent future AI will do something very clever like hide in a coding assistant and then distribute bits of itself around the planet in the source code of all of our stoplights and medical equipment.

That was part of the plot of person of interest. A really interesting show, it started as a basic "monster of the week" show but near the end it became a much more interesting plot.

Although most of the human characters were extremely one-dimensional. Especially Jim Caviezel's who was just a grumpy super soldier in every episode. It was kinda funny because they called him "the man in the suit" in the series and there was indeed little else to identify his character. The others were hardly better :(

But the AI storyline I found very interesting.

It won't be that simple because it will anticipate your every move once it gets intelligent enough.

Great analogy! I'll borrow this when explaining my thoughts on how LLMs pose to replace software engineers.

I tried replacing myself (coding hat) and it was pretty underwhelming. Some day maybe.

That comparison would make sense if the company was open source, non-profit, promised to make all designs available for free, took Elon Musk's money and then broke all promises includidng the one in its name and started competing with Musk.

> A bit of a "dog bites man" story to note that a CEO of a hot company is hyping the future beyond reason.

Why is it in your worldview a CEO “has to lie”?

Are you incapable of imagining one where a CEO is honest?

> The real story of LLMs is revealed when you posit a magical technology that can print any car part for free.

I’ll allow it if you stipulate that randomly and without reason when I ask for an alternator it prints me toy dinosaur.

> It is easy to imagine that the inventor of such a technology

As if the unethical sociopath TFA is about is any kind of, let alone the, inventor of genai.

> Being able to generate code/text/images is of no use to someone who doesn't know what to do with it.

Again, conveniently omitting the technology’s ever present failure modes.

A CEO has a fiduciary duty to lie.

A CEO that is the CEO of a publicly traded company who has over-leveraged their position and is now locked into chasing growth even if the company they're running suffers has a fiduciary duty to lie. There are plenty of CEOs not in this position.

I'm not talking about Altman in particular, I'm just annoyed with this constant spam on HN about how we all need to turn a blind eye to snake oil salesman because "that's just how it's supposed to be for a startup."

For a forum that complains about how money ruins everything, from the Unity scandal to OSS projects being sponsored and "tainted" by "evil companies," it's shocking to see how often golden boy executives are excused. I wish people had this energy for the much smaller companies trying to be profitable by raising subscriptions once in the 20 years they've been running, but instead they are treated like they burned a church. It truly is an elitist system.

Perhaps someone should have considered what problem they are trying to solve before spending vast resources on the "solution"?

There are plenty of plastic parts in cars and you can print them with 3D printer. I don't think that anything really changed because of that.

Because those are irrelevant to the point being made in the GP

GP? What happened to the good old "OP"?

OP is original poster (or post), to which we are not referring.

GP != OP; https://news.ycombinator.com/item?id=8363646

I think that is GP's point.

Reply on news.ycombinator.com