It’s hard to be disciplined about a black box though. That’s one reason why we’re all speeding off at a thousand miles per hour on transformers - the architecture works, why try other things?
Attention was invented because Bengio lab had to be disciplined about a black box (google had more compute)
whoever comes up with the next trick could win big.
Could, or meta makes you irrelevant with by reproducing your trick and giving it away for free
Yann LeCun certainly seems to be one of the more interesting people in the space. He’s a notable skeptic of Llm intelligence and incredibly smart.
To be fair I don't think anybody saw the boom in LLMs coming from the initial Attention paper. At the time it was one of many ideas that sounded like they had potential.
Whoever chooses the next winning lottery numbers could win big...
that's kinda my point, we don't know what the next winning move will be.