Hacker News Reader

Comment by popinman322 | Hacker News Reader

There was a discussion about the paper[0] this was based on 7 days ago: https://news.ycombinator.com/item?id=35768824

Glad it came up again. I missed it the first time. I am a complex systems scientist and its hard to overstate the importance of this.

The fact that there are reasonable metrics that make the emergence completely predictable as a simple threshold effect is massive.

One alternative way to phrase this that I haven't seen in either thread yet (but I am on mobile and haven't looked thoroughly) would be: Human observers/users dramatically underestimate how much small language models already have learned, because we are very good at spotting the errors. Scaling up looks like emergence occurs because the last few errors get eliminated.

If anything, it looks to me like the researchers are saying the opposite: We understand how this works, so it cannot be magic or special. The fact that you think it is means you are just wrong.

Sadly it is possible I have arrived at this conclusion due to previous experience with the academic world, which is ironically full of magical thinkers that will take any sort of AI advancement and scoff because it may mean they're no longer as special as they feel they must be. It is a very hostile environment.

They are saying correctly that we _can_ understand the learning of models in a continuous fashion, and that we do not need to assume that at some stage something different than at other stages happens. So no emergence.

This is massive because it means, for example, that you can study what is happening in these models with small ones that perform worse but are simpler to understand, and you're not missing something fundamental. I don't think they claim we already understand what is happening I these models though.

I don't know what exactly you are talking about with your academia bashing. If anything it seems to me academia is just as awash as everywhere else with barely substantiated AI hype. Disparaging and psychologizing critical voices just makes it seem to me like you might be missing the point of academia...

Great summary, I think it clarifies an important point. It's not that these models aren't able to do the surprising things people are claiming they can do, but that in fact the smaller versions of the models had these type of capabilities as well, but to a lesser degree, and nobody was aware of it because we weren't even looking for it.

what does a complex systems scientist do?

See the world from a systemic relational approach, and publish and talk about it.

Thanks! Macroexpanded:

Are emergent abilities of large language models a mirage? - https://news.ycombinator.com/item?id=35768824 - May 2023 (126 comments)

Reply on news.ycombinator.com