cgadski

But it _is_ possible to make a ball out of hexagons if you use infinitely many of them!

In my experience, the discrete version of the Gauss-Bonnet theorem is a really great conversation starter to talk about the kinds of things mathematicians like. Everyone knows that the exterior angles in a polygon has to sum up to 2 pi, but did you know that you can define "exterior angles" on the vertices of a polyhedron that have to sum up to 4 pi? Can you think of a single line of reasoning that explains both facts? (Remember that the circumference of the unit circle is 2 pi and the area of the unit sphere is 4 pi...)

That's a neat trick, "cancelling" out catastrophic cancellation. Of course it doesn't work anymore once x is smaller than machine epsilon, whereas expm1(x) / x will continue to work.

In general herbie is pretty good at suggesting the right functions / rearrangements. For this example, it finds expm1: https://herbie.uwplse.org/demo/378a0682ec4d3e735c790a58fa97e...

Dynamical systems definitely don't have to preserve volume!

It's great to see so much interest in this post on HN! In view of the attention, I feel like I need to emphasize that although my writing and illustrations are original, the "big idea" certainly is not. In fact, my goal was just to present the simplest example I could find that illustrates a connection between gradient descent and kernel methods. There's already a lot of work being done on this relationship, most of which I'm just now learning of.

(Although I don't have time to respond meaningfully today, I really appreciate the comments pointing out other relationships by mvcalder, Joschkabraun, yagyu and others.)

Yes indeed, my blog stylesheet started out as water.css. It was really helpful to get good typesetting and color defaults.

My guess would be that I have to take another look at optimizing the javascript I use for drawing the interactive widgets. It's just bare-bones javascript and canvas, but a previous version managed to freeze some browsers for unknown reasons...

Took a preliminary look through and it feels pretty exciting. Thanks for sharing.

My own post basically covers the simplest possible case where gradient descent does something related to kernels. The problem is that the "tangent kernel" driving the evolution of the model over training is typically not constant. (In my case it is constant because my model is linear.)

Domingos' solution seems to be: in general, just integrate the tangent kernel over the path taken by your optimization and call it the path kernel. Then your resulting model can always be viewed as a kernel machine, with the subtlety that the kernel now depends on the trajectory you took during training. So in that sense, kernels are everywhere! I'll take another look on Monday :)

I had Chapter 6 open while working on this post :) I should probably cite it too. As far as I could see that book doesn't talk about kernels arising from gradient descent, but it does say many other things about GP kernels. Certainly it has the main things I used.

It's amusing how facts that look surprising and mysterious to the rest of the world are just table stakes at the right sort of math department. As a researcher I feel the pressure to make things that are "my own", but there's so much that already exists just waiting to be grokked and plugged in!

As the author of that other post, I'm happy to see Terry Tao's post on the front page now too. I learned a lot from it when I was researching for my own. The two articles end up going in somewhat different directions---Terry doesn't talk about dimensions with complex exponents, for example---but his is definitely more complete.

My favorite bit is his explanation of how dimensional analysis explains the relationships between powers used in Sobolev inequalities. After we get the idea of dimensions divorced from the usual physical MLT usage you can apply these techniques all over math.

In the formalism of both this article and https://news.ycombinator.com/item?id=37517118, "base units" are specially designated physical quantities, while "dimensions" describe how physical quantities scale under changes to the scale of measurement.

Customarily, capital letters like M, T, L, etc., refer to dimensions, and you would something like [1 kg] = M to say that 1 kg is a quantity that scales like a mass. A PLT analogy analogy would be to compare physical quantities to values and dimensions to types.

In practice we can refer to dimensions by base units, but it has some mathematical benefits to view them as distinct. Hope that clarifies things!

"Dimensions" in dimensional analysis are not the same thing as dimensions in linear algebra! I'm not sure what a scalar quantity with an infinite dimension would be. (To reiterate, the dimensions of dimensional analysis tell you how quantities scale under adjustment of our scale of measurement, so having infinite dimension would mean scaling infinitely quickly…?)

That being said, I did try to connect infinite-dimensional problems with dimensional analysis in this post. Towards the end of my section on dimensions with complex powers, I applied dimensional analysis to the study of periodic signals and ended up arguing that (complex) Fourier basis elements can be thought of as dimensionful objects. I haven't seen dimensions used like this before, so I was hoping to earn the vote from fans of functional analysis ;)

I had two design goals for my tiny personal page:

  * Write the content in Markdown.
  * Don't require any source files to update it.

And here we are: https://cgad.ski.