Hacker News Reader

Comment by extr | Hacker News Reader

IMO if anything I am coming to the opposite conclusion. Yud and his entire project failed to predict literally anything about how LLMs work. So why take anything they say seriously? Can anyone name a single meaningful contribution they've made to AI research? The whole thing has been revealed to be crank science. At this point it seems like they will continue to move goalposts and the AI superintelligence apocalypse will be just around the corner until one day we will wake up and LLMs or their descendents will be integrated into everyday life and it will be totally fine.

View on HN

LLMs have gained emergent new abilities at every generation. No one programmed those in or even expected them to appear, they arose spontaneously with larger computation resources. "The AI developed new abilities we didn't intend or anticipate" sure sounds to me like the kinds of dangers AI alignment researchers have been theorizing for years.

We've only begun to build multimodal systems; GPT-4 already exhibits improved ability when trained with images vs without.

It isn't "crank science" at all. Despite their mockery and personal insults and dismissive handwaving, no AI capability-pusher has yet made a convincing argument why an entity with extremely strong cognitive powers would not be capable of transforming its surrounding environment on unprecedented scale. The closest one we've heard is "we just won't be able to make AI very smart." Which wishfully will be true, but we're about to spin up multiple Apollo-programs trying to prove that it isn't.

Do you have a counterexample of someone who's gotten their predictions right? Because if not, that should only terrify you even more. If there's anyone out there who predicted how LLMs would work in a way Eliezer failed to, and if that person is predicting "AGI will be cool and will naturally prioritize our well-being", I would love to know.

>Do you have a counterexample of someone who's gotten their predictions right? Because if not, that should only terrify you even more. If there's anyone out there who predicted how LLMs would work in a way Eliezer failed to, and if that person is predicting "AGI will be cool and will naturally prioritize our well-being", I would love to know.

A recent interview with Paul Christiano is about the closest I've come to this. He does note some semi-accurate predictions at the linked timestamp, but the forecast for how things are likely to go is not exactly rosy, though he's quite a bit more optimistic than Eliezer.

https://youtu.be/GyFkWb903aU?t=1357

Also this whole interview was pretty interesting. Near the end he details how few people world-wide actually work on X-risk from AGI. He also outlines how the academic ML community in general just continually keeps getting predictions really wrong, and many aren't taking X-risk seriously.

Overall his is the most balanced take I've seen. A lot better than Eliezer.

I'll be glad to check it out later, but:

> he details how few people world-wide actually work on X-risk from AGI ... and many aren't taking X-risk seriously

still sounds extremely dangerous.

It's a fantastic interview. To clarify, his position boils down to something like "the risk is very real, the chance the risk materializes is quite significant, it's not a foregone conclusion that we are all definitely going to die from it, so there is some hope, but it's likely the default outcome and there aren't many people who take it seriously or who are working on it." I would say he's net bearish on it, which is why he spends time working on trying to mitigate the X-risk, or attempting to at least.

>IMO if anything I am coming to the opposite conclusion. Yud and his entire project failed to predict literally anything about how LLMs work.

Or you could take that as evidence (and there's a lot more like it) that AGI is a phenomenon so complex that not even the experts have a clue what's actually going to happen. And yet they are barrelling towards it. There's no reason to expect that anyone will be able to be in control of a situation that nobody on earth even understands.

After watching virtually every long-form interview of AI experts I noticed they each have some glaring holes in their mental models of reality. If even the experts are suffering from severe limitations on their bounded-rationality, then lay people pretty much don't stand a chance at trying to reason about this. But let's all play with the shiny new tech, right?

What have they been wrong about?

What have they been right about is a much shorter list.

> Yud and his entire project failed to predict literally anything about how LLMs work.

In their defense, LLMs did come a bit out of the blue. In retrospect, Yudkowsky and his disciples were focusing too much on rationality as science/mathematics, bending Bayes to the point of breaking to try and gleam how perfect intelligence works, and how to somehow formalize the aggregate mess of our fuzzy morality.

They failed to predict that shoving gigabytes of random Internet text at a NN, and having it place parts of words as points in a hundred thousand dimension vector space, will suddenly reduce most of what we consider "thinking" into proximity search in said vector space. They were so focused on the theory, they failed to predict the brute-force, messy practice. But so did everyone else.

If anything, Yudkowsky & co. were the only people consistently taking the problem seriously, and got the outline right.

Reply on news.ycombinator.com