Hacker News Reader

That's not completely true. The architecture must behave well for scaling, which is not trivial. Basic multi-layer perceptrons do not scale well for example, the gradient will vanish or explode deeper in the network.

View on HN

And data quality. Ensuring the sourcing and quality is very important to get a good model.

this, if you have money to spend in improving your model, more training data is the first thing I'd take a look at

How do modern foundation models avoid multi-layer perceptron scaling issues? Don't they have big feed-forward components in addition to the transformers?

They don't do global optimisation of all layers at the same time, instead training all layers independently of each other.

They rely heavily on what we call residual or skip connexions. This means each layer does something like x = x + f(x). This helps the training a lot, ensuring the gradient can flow nicely in the whole network.

This is heavily used in ResNets (residual networks) for computer vision, and is what allows training much deeper convolutional networks. And transformers use the same trick.

Reply on news.ycombinator.com