I've only skimmed the article, but the idea looks to be "Kernel methods" [0] (and showing their equivalence in the infinite middle layer width limit?).

If anyone has more insight into the actual idea, I would appreciate an explanation.

In the article, there are two advertisements to older articles. One is "New Theory Cracks Open the Black Box of Deep Learning" from 2017 and the other is "Foundations Built for a General Theory of Neural Networks" from 2019.

[0] https://www.quantamagazine.org/a-new-link-to-an-old-model-co...