AlphaGo Moment for Model Architecture Discovery
arxiv.org
arxiv.org
They discovered 106 new state-of-the-art linear attention architectures through a fully autonomous AI research loop. The authors are making comparisons to AlphaGo’s move 37.
If it were it would have the on of the worst title imaginable and include one of the worst abstracts for any such paper.
It seriously is a very big red flag that a paper is doing this much to talk about how important it is. It is also essentially pre-writing the headlines for journalists to use, making this whole thing, at least, look like it is a PR stunt.
The one thing I didn't see that would be good is some validation that the architecture(s) that perform best on large models are the same architectures that perform best on small models.
Ie validation the assumption that you can use small models with sma amounts of training/compute to determine the best architecture for large models and high training budgets.
Even if it doesn't translate it would still be very cool to be able to qui kly evolve better small models (1M to 400M params), but I believe the implied goal (and what everyone wants) is that this exploration and discovery of novel architectures would be applicable for the really big models as well.
If you could only ai discover larger models by spending OpenAi/Anthropic/... budgets per exploration then we're not really gaining much in terms of novel ideas as the cost (time and budget) would be too prohibitive.