Hacker News Reader

Comment by DeathArrow | Hacker News Reader

Architecture doesn't make a difference. Big enough models trained with big enough data tend to give the same results regardless of architecture. So yes, most advances in AI are mostly due to the fact we can now multiply matrices very fast.

View on HN

That's not completely true. The architecture must behave well for scaling, which is not trivial. Basic multi-layer perceptrons do not scale well for example, the gradient will vanish or explode deeper in the network.

And data quality. Ensuring the sourcing and quality is very important to get a good model.

this, if you have money to spend in improving your model, more training data is the first thing I'd take a look at

How do modern foundation models avoid multi-layer perceptron scaling issues? Don't they have big feed-forward components in addition to the transformers?

They rely heavily on what we call residual or skip connexions. This means each layer does something like x = x + f(x). This helps the training a lot, ensuring the gradient can flow nicely in the whole network.

This is heavily used in ResNets (residual networks) for computer vision, and is what allows training much deeper convolutional networks. And transformers use the same trick.

They don't do global optimisation of all layers at the same time, instead training all layers independently of each other.

idk, they do give the same results, but given the memory bottleneck it feels like we are at a point when architecture innovations matter again, for example check out DeepSeek V2 tech report, they modded model arch specifically for lower cost inference (by making k/v cache smaller)

Different architecture can result in hundreds of millions of dollars more in training costs no?

Sure, but the point wasn't about the costs, it was about the capability of a trained model.

Reply on news.ycombinator.com