Hacker News Reader

My summary: Anything which drives predictions of "the next token" must create a model of "how the world works" AND a model for "what state is the world in right now". The authors train a transformer and demonstrate that it creates a structure that represents both of these.

View on HN

Crucially it will tend to find the simplest such representation that still solves the problem. This is why ultimately the model is only sufficient to solve problems that it was trained to solve.

Simplest in terms of optimization, by the way.

What do you mean by simplest in term of optimization?

I get it find solution that are easy for SGD or Adam optimizer to find.

But why would such solution be less simple than other ?

(I could be wrong here. Please correct me if that's the case.)

I think the comment you're replying to means exactly what you're saying, which is that it will find solutions which are "easy" to find for the optimizer, and therefore solutions which are simple to achieve through the convergence of some optimizer.

Wonder if it can be applied to financial markets and make short term options trading profitable.

Reply on news.ycombinator.com