after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.
The rest of us reading this are like, holy batman, what the fuck was that?!
You could have your own hacker news, it's just a textbox, a bunch of tables and headings! Once you add these, it'll be abundantly clear that you also need a database. It should now be obvious that you also need a user system and it can be easily shown that needs a backend. Admin tools, tests, statistics, performance checks and so on can easily be derived from such backend.
A piece like this which uses it in a headline but in no way makes an average reader feel like they could have come up with it is just badly misjudging how good of an explanation it is.
“You Could Have Invented Parser Combinators”
“You Could Have Invented Container Runtimes”
“You Could Have Invented Git”
Given the references to “mathematicians”, I think this reaction is more about an unfamiliarity with the concept of applied mathematics, which is ironic for practitioners in a field containing so much that is (or should be) regarded that way.
Software used to be all about “discrete math”, logic, maybe a bit of queuing theory…and now suddenly linear algebra and statistics became important. Don’t panic, it’s just another textbook on the shelf.
What this particular mathematical writeup seems to address is ‘how does someone who is already up to speed on the mathematical underpinnings of attention kernel neural networks get brought up to date on what they need to understand in order to see this new algorithm as obvious’
Which is a smaller audience to be sure, and - given how fast this field has grown - somewhat disingenuous in its implications.
This kind of title is appropriate for a piece aimed at helping someone who is a little behind the curve catch up. In the field of deep learning it should be obvious even to the people inside it riding that curve that it is moving really fast and that this kind of content only works for people who are running to keep up.
[1] https://medium.com/@gtrevorjay/you-could-have-invented-conta...
I often find people get annoyed at mathy stuff because they seem to think that they should be able to read it like a (comparatively low information dense) newspaper article or something similar.
Math isn't like that, it usually has high information density and you need to parse every single symbol. And also people make this mistake where they gloss over stuff they don't get because they think they'll just understand things from context. Works great in normal literature - but math ain't like that. If you don't understand something, go back to the definitions.
Writing a detailed technical post behind the news of Kimi K3 and KDA algorithm with an audacious title like "You Could Have Invent Breakthrough It too" they are pre-filtering out the ones who couldn't comprehend with quick read (myself included) and attracting the ones who agreed with the blog post. At the end with a strong CTA to promoting their 10x cheapter open weight model AI inference and hiring too.
Good job Doubleword, I see what you are doing there.
We're just a group of guys and gals who like inference!
Nothing complex
"New algorithm, or fmadd?"
"... fmadd."
"You could have come up with Kimi Delta Attention, but you didn't, did you."