It's great to see so much interest in this post on HN! In view of the attention, I feel like I need to emphasize that although my writing and illustrations are original, the "big idea" certainly is not. In fact, my goal was just to present the simplest example I could find that illustrates a connection between gradient descent and kernel methods. There's already a lot of work being done on this relationship, most of which I'm just now learning of.

(Although I don't have time to respond meaningfully today, I really appreciate the comments pointing out other relationships by mvcalder, Joschkabraun, yagyu and others.)