Counterargument: this blogpost is worthless. You get all the way to the end and then find out he hasn't actually tried it, not even on a toy model. It's just a neat idea he thinks will work.
Explainers and their folksy, imprecise tone are good for things we already know are true. I’m skeptical on things which are unproven.
There are a thousand papers out there making minor tweaks to the transformer architecture. 99% of them are also worthless and forgotten.
That's precisely what he shared this for, though. So someone willing to train a model with this tweak tries it.
Ideas like sparse attention, tree attention, residual attention, etc, all sound good on paper, but when researchers try to reproduce them they either find no results or results that don't scale. Even AliBi is turning out to be less powerful than scaled-down positional embeddings. It's almost a bitter lesson on its own: you can't beat the original transformer.
Optimizations that do stick around tend to be the ones that preserve the original algorithm but help with caching or memory accesses.
With large ML models, there probably is no intuition like this. We just don't know "if I do the common sense thing X, it surely will produce better results for a given benchmark" ... well we have no idea until it is tried out.
> I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead.
So I think your accusation of his burying the lede on the lack of experiment is unwarranted.