Comment by toss1 | Hacker News Reader

Comment by toss1 | Hacker News Reader

Parent Full thread

There is no question LLMs are truly useful in some areas, and the LLM bubble will inevitably burst. Both can be simultaneously true, and we're just running up the big first slope on the hype curve [0].

As we learn more about the capabilities and limits of LLMs, I see no serious arguments scaling up LLMs with increasingly massive data centers and training will actually reach anything like breakthrough to AGI or even anything beyond the magnitude of usefulness already available. Quite the opposite — most experts argue fundamental breakthroughs will be needed in different areas to yield orders-of-magnitude greater utility, nevermind yielding AGI (not that much more refinement won't yield useful results, only that it won't break out).

So one question is timing — When will the crash come?

The next is, how can we collect in an open and preferable independently/distributed/locally-usable way the best usable models to retain access to the tech when the VC-funded data centers shut down?

[0] https://en.wikipedia.org/wiki/Gartner_hype_cycle

We even have prior art. Web 1.0 and e-Commerce were truly useful and the bubble also burst.

I should also think further, railroads and radio also good examples!

Yes well bubbles are a core part of the innovation process (new tech being useful doesn't imply a lack of bubbles), see e.g."Technological Revolutions and Financial Capital" by Carlota Perez https://en.wikipedia.org/wiki/Technological_Revolutions_and_...

Unlike that time, some money is actually being made. I heard some figures thrown around yesterday, total combined investments of over 500 billion! and revenues of about 30 billion, 10 bil of which was payments to cloud providers, so actually 20 billion in revenues. that's not nothing.

Plenty of e-commerce places had revenue, they just didn't have profit and they usually spent on crazy stuff, like Super Bowl ads.

It might not be a paradox: Bubbles are most likely to occur when something is plausibly valuable.

If GenAI really was just a "glorified autocorrect", a "stochastic parrot", etc, it would be much easier to deflate AI Booster claims and contextualise what it is and isn't good at.

Instead, LLMs exist in a blurry space where they are sometimes genuinely decent, occasionally completely broken, and often subtly wrong in ways not obvious to their users. That uncertainty is what breeds FOMO and hype in the investor class.

I use LLMs all the time and do ML and stuff. But at the same time, they are literally averaging the internet, approximately. I think the terms glorified autocomplete and stochastic parrot describe how they work under the hood really well.

A top expert in US Trust & Estate Tax law whom I know well tells me that although their firm is pushing use of LLMs, and they are useful for some things, there are serious limitations.

In the world of T&E law, there are a lot of mediocre (to be kind) attorneys who claim expertise but are very bad at it (causing a lot of work for the more serious firms and a lot of costs & losses for the intended heirs). They often write papers for marketing themselves as experts, so the internet is flooded with many papers giving advice that is exactly wrong and much more that is wrong in more subtle ways that will blow up decades later.

If an LLM could reason, it would be able to sort out the wrong nonsense from the real expertise by applying reason, e.g., comparing the advice to the actual legal code and precedent-setting rulings, and by comparing it to results, and be able to identify the real experts, and generate output based on the writings of the real experts only.

However, LLMs show zero sign of any similar reasoning. They simply output something resembling the average of all the dreck of the mediocre-minus attorneys posting blogs.

I'm not saying this could not be fixed by Altman et. al. applying a large amount of computer power to exactly the loops I described above (check legal advice against the actual code and judges' rulings, check against actual results, select only the credible sources and retrain), but it is obviously no where near that yet.

The big problem, is that this is only obvious to a top expert in the field who deeply knows from training and experience the difference between the top experts and the dreck.

To the rest of us who actually need the advice, the LLMs sound great.

Very smart parrot, but still dumbly averaging and stochastic.

Yup, I find LLMs are fantastic for surfacing all kinds of "middle of the road" information that is common and well-documented. So, for getting up to speed or extracting particular answers about a field of knowledge with which I'm unfamiliar, LLMs are wonderfully helpful. Even using later ChatGPT versions for tech support on software often works very well.

And the conversational style makes it all look like good reasoning.

But as soon as the wanders off the highways into little-used areas of knowledge (such as wiring for a CNC machine controller board instead of a software package with millions of users' forum posts), even pre-stuffing the context with heaps of specifically relevant documents rapidly reveals there is zero reasoning happening.

Similarly, the occasional excursions into completely the wrong field even with a detailed prompt show that the LLM really does not have a clue what it is 'reasoning' about. Even with thinking, multiple steps, etc., the 'stochastic parrot' moniker remains applicable — a very damn smart parrot, but still.

Reply on news.ycombinator.com