Hacker News Reader

Comment by gyomu | Hacker News Reader

The reasoning in the article is interesting, but this struck me as a weird example to choose:

> “The real question is how can we predict when a new LLM will achieve some new capability X. For example, X = “Write a short story that resonates with the social mood of the present time and is a runaway hit”

Framing a capability as something that is objectively measurable (“able to perform math on the 12th grade level”, “able to write a coherent, novel text without spelling/grammar mistakes”) makes sense within the context of what the author is trying to demonstrate.

But the social proof aspect (“is a runaway hit”) feels orthogonal to it? Things can be runaway hits for social factors independently of the capability they actually represent.

View on HN

It’s not about being “a runaway hit” as an objective measurement it’s about the things an LLM would need to achieve before that was possible. At first AI scores on existing tests seemed like a useful metric. However, tests designed for humans make specific assumptions that don’t apply to these systems making such tests useless.

AI is very good at gaming metrics so it’s difficult to list some criteria where achieving it is meaningful. A hypothetical coherent novel without spelling/grammar mistakes could in effect be a copy of some existing work that shows up in its corpus, however a hit requires more than a reskinned story.

> however a hit requires more than a reskinned story.

demonstrably false with a lot of hits in the past that is a reskinned story of existing stories!

While not a technical term of art, copyright applies to a reskinned story. “the series is not available in English translation, because of the first book having been judged a breach of copyright.” https://en.wikipedia.org/wiki/Tanya_Grotter

There’s plenty of room to take inspiration and go in another direction aka Pride and Prejudice and Zombies.

That it seems hard (impossible) or not clear intuitively how to go about it, to us humans, is what makes the question interesting. In a way. The other questions are interesting but a different class of interesting. At any rate, both good for this question. Either way this becomes "what would we need to estimate this emergence threshold?".

Reply on news.ycombinator.com