Is next token prediction the new "E = mc2"? | Hacker News Reader

Is next token prediction the new "E = mc2"?

4 pointsdbb_io7 comments

History has taught us that the most significant discoveries of formulas are often those that seem to be the simplest equations (even seem too simplistic to be true) but generalize the most.

Take Einstein's E = mc2 as an example.

My (maybe purely philosophical) hypothesis is now: Next token prediction might be for AI, what "E = mc2" has been for physics.

When Ilya Sutskever was asked in an interview what surprised in the most regarding the rapid advances in AI, his answer was, that "it" (meaning large language models and especially next token prediction) works at all.

And I agree. While it is easy to understand what next token prediction is doing, it is mindblowing what outputs can be achieved with such simplistic formula.

I'd like to discuss - do you think it's a valid statement to say "Next token prediction is the new E = mc2"?

It feels very strange to put "next token prediction" on the same level as "E = mc2".

"next token prediction" is an experiment that's puzzling, while E=mc2 is a theory (that Einstein developed to explain the puzzling experiments that were unexplained at the time).

It may or may not be that in the end there is a "simple formula" that explains why LLMs work, but we're not there yet.

Basic markov chain models are also next token prediction, so it isn't new, we have done that to make language models for over half a century at this point. It is one of the earliest things people did with computers.

Our modern LLMs are an evolution of that, but the next token prediction part is almost as old as computers.

The real invention is having a model that converges, does not lose the gradients, and having a framework to reliably train lots of parameters. Imo it is not historically accurate that next token prediction is the big invention. Some language models predict blanked tokens in the middle of sentences that are modern and interesting, and old language models like rnn-s are being used for quite some time, predicting next tokens one by one.

No. It's on the same level of categorical error as "A pumpkin is the new hedgehog."

It took 40 years to get from E=mc^2 to the Trinity test, so ask again in the early 60s

That sequence of events had a lot of catalyst!

There’s plenty of time for catalysts to show up before the 60s get here.

My point is really just that it’s too soon to tell.

Reply on news.ycombinator.com