Hacker News Reader

Comment by libraryofbabel | Hacker News Reader

Way back when, I did a masters in physics. I learned a lot of math: vectors, a ton of linear algebra, thermodynamics (aka entropy), multi-variable and then tensor calculus.

This all turned out to be mostly irrelevant in my subsequent programming career.

Then LLMs came along and I wanted to learn how they work. Suddenly the physics training is directly useful again! Backprop is one big tensor calculus calculation, minimizing… entropy! Everything is matrix multiplications. Things are actually differentiable, unlike most of the rest of computer science.

It’s fun using this stuff again. All but the tensor calculus on curved spacetime, I haven’t had to reach for that yet.

View on HN

Check out this 156-page tome: https://arxiv.org/abs/2104.13478: "Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges"

The intro says that it "...serves a dual purpose: on one hand, it provides a common mathematical framework to study the most successful neural network architectures, such as CNNs, RNNs, GNNs, and Transformers. On the other hand, it gives a constructive procedure to incorporate prior physical knowledge into neural architectures and provide principled way to build future architectures yet to be invented."

Working all the way through that, besides relearning a lot of my undergrad EE math (some time in the previous century), I learned a whole new bunch of differential geometry that will help next time I open a General Relativity book for fun.

I have very little formal education in advanced maths, but I’m highly motivated to learn the math needed to understand AI. Should i take a stab at parsing through and trying to understand this paper (maybe even using AI to help, heh) or would that be counter-productive from the get-go and I'm better off spending my time following some structured courses in pre-requisite maths before trying to understand these research papers?

Thank you for sharing this paper!

You might take a course on linear-algebra, at K.A. for example:

https://www.khanacademy.org/math/linear-algebra

And any prereqs you need. I also find the math-is-fun site to be excellent when I need to brush up on something from long ago and want a concise explanation. i.e. A 10 minute review, more than a few pithy sentences, yet less than a dozen-hour diatribe.

https://www.mathsisfun.com/

Thank you for sharing this!

Thank you for sharing the paper!

The link is broken though and you may want to remove the `:` at the end.

That past work will pay off even more when you start looking into diffusion and flow-based models for generating images, videos, and sometimes text.

Breakthrough in image generation speed literally came from applying better differential equations for diffusion taken from statistical mechanics physics papers:

https://youtu.be/iv-5mZ_9CPY

For me, it's the very basics of general relativity which made the distinction between the cotangent and tangents space click. Optimisation on Riemannian manifolds might give an opportunity to apply more interesting tensor calculus with a non-trivial metric.

I have the same experience but with a masters in control theory. Suddenly all the linear algebra and differential equations are super useful in understanding this.

Any reason you didn't pick up computer graphics before? Everything is linear algebra and there's even actual physics involved.

Is that you Will Hunting

Well, calculus on curved space, please let me introduce you to: https://arxiv.org/abs/2505.18230 (This is self advertising) If you know how to incorporate time into that, I am interested.

The funny thing about physics maths, is that we didn't have to learn the intuition behind it, it was a mean to an end. Going through undergrad mathematically blind was a right of passage.

Modern numeric compute frameworks provide automatic differentiation to calculate derivatives, including Tensorflow and Jax.

Reply on news.ycombinator.com