Hacker News Reader

Comment by joe_the_user | Hacker News Reader

After the failure of FB's Galactica model, I wonder whether LLM are starting to hit diminishing return. I've read of several ways diminishing returns could happen:

* Not enough training data - you've used up the Internet (even a percentage of the Internet might be as much as is usable by clever brute force).

* Not enough compute time to fully train (we're not close to that)

* The model covers such a large area that testing is impossible

One thing I'd speculate about is perhaps the more different subjects the program is expected to combine, the more it learns to spout plausible bullshit and clever quips, since for clever humans, that how they relate to stuff they don't know. So "pretentious but uninformed" might be a sign.

View on HN

It's also likely we're just seeing the limits of what the tech can do. It's not actually "intelligent" it's effectively a cool trick, and at some point the gap between what the trick can do and what a "perfect" AI can do becomes obvious. It's really not that different from Eliza being cool for a few minutes before being obviously found wanting. It just takes longer for the shine to wear off as we get to a better technology. Ditto for self driving cars - a steep climb to a 90% solution, then an impossible chasm to cross to something that actually handles the edge cases.

It’s also not trained the way humans are. We’re trained interactively, and from an early age, say things in order to express ourselves and try to get what we want. Machine learning models are generally trained passively.

The chatbots using the model over at character.ai give you a sort of genetic algorithmic interface by letting you choose from alternative responses and vote on them as conversations progress, and you can make your own bots then train them and get some very interesting results. I am personally extremely optimistic about the future of such technology and I think we've only begun to scratch the surface collectively.

I refuse to believe that anybody who compares LLMs to Eliza has interacted with the technology for more than 10 minutes.

I think the poster's point was that initial reactions to LLMs are similar to initial reactions to Eliza. Everyone was gobsmacked for a while, but then once the shine wore off and its limitations were exposed, everyone got bored and moved on quickly.

Ah yes, the AI Effect in action.

https://en.wikipedia.org/wiki/AI_effect

I am eagerly awaiting GPT4, it might be the case that language models will hit a plateau, just like video games graphics hit a plateau too, with each generation being less and less impressive compared to the previous, while still not being perfect

That’s an odd comparison since unreal engine 4 is incredibly close to live action. I watched the recent matrix demo several times in awe.

I still need to read past the first page, but I was reminded this morning of the 2021 paper “On the dangers of stochastic parrots: can language models be too big?” which discusses the harms and diminishing returns of moving to ever larger language models.

I had to rush out the door today after seeing this paper come up so I can’t speak much to its content right now. But if anyone wants to read it and reflect here I’d like to hear it.

https://dl.acm.org/doi/10.1145/3442188.3445922

Reply on news.ycombinator.com