Hacker News Reader

There is evidence that people believe that GTP-4 is intelligent since it can solve things like the SATs. But if you start taking away weights one by one at some point those same people will say it isn't intelligent. A NN with 3 weights cannot solve any problems that humans believe requires intelligence. So where did it come from? I don't know, but it clearly emerged as the NN got bigger.

View on HN

It's possible that the comment you're responding to is referring to the new paper out of Stanford/Google on emergence as a 'mirage'. The gist of it is that the shape of the curve for 'metric' vs 'params' is dependent upon the type of metric. For classification it has a certain shape, for accuracy it has another, etc. The authors claim the observation that these curves are determined by the choice of metric means that it isn't true emergence, but rather is due to the way in which GPTs are typically trained to predict the next token (and then beam search or typical sampling is often applied afterwards, etc).

The paper is somewhat new so I haven't done a proper review to know if it's solid work yet, but this may offer some context for some of the comments in this thread.

Reply on news.ycombinator.com