Hacker News Reader

Comment by lz400 | Hacker News Reader

Engineers thinking they're building god is such a good marketing strategy. I can't overstate it. It's even difficult to be rational about it. I don't actually believe it's true, I think it's pure hype and LLMs won't even approximate AGI. But this idea is sort of half-immune to criticism or skepticism: you can always respond with "but what if it's true?". The stakes are so high that the potentially infinite payoff snowballs over any probabilities. 0.00001% multiplied by infinite is an infinite EV so you have to treat it like that. Best marketing, it writes itself.

View on HN

Similar to Pascal's wager, which pretty much amounts to "yeah, God is probably not real, _but what if it is_? The utility of getting into heaven is infinite (and hell is infinitely negative), so any non-zero probability that God is real should make you be religious, just in case."

https://en.wikipedia.org/wiki/Pascal%27s_wager#Analysis_with...

I am convinced!

Which god should I believe in, though? There are so many.

And what if I pick the wrong god?

You should believe in all of them. Just spray and pray!

You have to filter out all the religions where you can't worship other gods or be in other religions.

Ah, so all of them. Oh well!

Might as well skip the one that don't punish you for not believing as well.

Any God, as long as it's not a being/creature. Thankfully there's only one of those! https://en.wikipedia.org/wiki/Classical_theism

See also Pascal's mugging, from Eliezer Yudkowsky. Some would say AI Safety research is a form of Pascal's mugging.

https://en.wikipedia.org/wiki/Pascal%27s_mugging

This is explicitly not the conclusion Pascal drew with the wager, as described in the next section of the Wikipedia article: "Pascal's intent was not to provide an argument to convince atheists to believe, but (a) to show the fallacy of attempting to use logical reasoning to prove or disprove God..."

Did he say Pascal drew that conclusion and remove it with an edit or something? As it's written now it seems like you're correcting him for something he didn't post.

I know you're not being serious but building AGI as in something that thinks like a human, as proven possible by millions of humans wandering all over the place is very different from "building god".

Except that humans cannot read millions of books (if not all books ever published) and keep track of massive amounts of information. AGI presuposes some kind of super human capabilities that no one human has. Whether that's ever accomplished remains to be seen, I personally am a bit skeptical that it will hapen in our lifetime but think it's possible in the future.

Not sure about that one. I do agree with the AI bros that, _if_ we build AGI, ASI looks inevitable shortly after, at least a "soft ASI". Because something with the agency of a human but all the knowledge of the world at its fingertips, the ability to replicate itself, think at order of magnitudes faster and paralelly on many things at the same time and modify itself... really looks like it won't stay comparable to a baseline human for long.

Though even if smarter than us we may get Sheldon off Big Bang Theory more than god.

> I don't actually believe it's true, I think it's pure hype and LLMs won't even approximate AGI.

Not sure how you can say this so confidently. Many would argue they're already pretty close, at least on a short time horizon.

Many would argue that you should give them a billion dollars funding, and that’s what they’re doing when they say AGI is close.

There is a decade + worth of implementation details and new techniques to invent before we have something functionally equivalent to Jarvis.

I mean, they're wrong? LLMs don't have agency, don't learn, don't do anything except react to prompts really.

What agentic tools have you tried?

"but what if it's true?"

There was nothing hypothesized about next-token prediction and emergent properties (they didn't know scale would allow it to generalize for sure). What if it's true is part of LLMs story, there is a mystical element here.

> There was nothing hypothesized that next-token prediction and scale could show emergent properties.

Nobody ever hypothesized it before it happened? Hard to believe.

Someone else can confirm, but from my understanding, no they did not know sentiment analysis, reasoning, few shot learning, chain of thought, etc would emerge at scale. Sentiment analysis was one of the first things they noticed a scaled up model could generalize. Remember, all they were trying to do was get better at next-token prediction, there was no concrete idea to achieve "instruction following", for example. We can never truly say going up another order of magnitude on the number of params won't achieve something (it could, for reasons unknown, just like before).

It is somewhat parallel to the story of Columbus looking for India but ending up in America.

> sentiment analysis, reasoning, few shot learning, chain of thought, etc would emerge at scale

Some would say it still hasn't (to an agreeable level).

The Schaeffer et al. "Mirage" paper showed that many claimed emergent abilities disappear when you use different metrics, what looked like sudden capability jumps were often artifacts of using harsh/discontinuous measurements rather than smooth ones.

But I'd go further: even abilities that do appear "emergent" often aren't that mysterious when you consider the training data. Take instruction following - it seems magical that models can suddenly follow instructions they weren't explicitly trained for, but modern LLMs are trained on massive instruction-following datasets (RLHF, constitutional AI, etc.). The model is literally predicting what it was trained on. Same with chain-of-thought reasoning - these models have seen millions of examples of step-by-step reasoning in their training data.

The real question isn't whether these abilities are "emergent" but whether we're measuring the right things and being honest about what our training data contains. A lot of seemingly surprising capabilities become much less surprising when you audit what was actually in the training corpus.

Didn't it just get better at next token prediction? I don't think anything emerged in the model itself, what was surprising is how really good next token prediction itself is at predicting all kind of other things no?

100%

Reply on news.ycombinator.com