Hacker News Reader

It's a kind of an open secret that there's no 'training' protocol for these state of the art models.

Researchers behave like alchemists when training these models, and the actions are not really reproducible.

They could provide access to the training code. It's useful for training smaller models or distilling larger ones. They don't need to release every details involved in tuning the optimization parameters during the pre-training stage.

There is no training 'code' that will get you anything close to a usable result.

Reply on news.ycombinator.com