Hacker News Reader

Comment by mjburgess | Hacker News Reader

The "math" used in ML/AI papers is usually just a sort of 'whiteboard math' which is a domain-specific mishmash of linear algebra, calculus, set theory and statistics.

If you could find a book just going through the relevant bits you wouldnt really have to "learn math again", it can be translated into english straightforwardly -- very very few ML papers relevant to industry have extended proofs, etc. that require eg., even being able to differentiate anything yourself.

90% of it is: here's the domain (ie., type) of our variables, here's the formula of our functions, we're taking a weighted average with some inner products involved.

It might sound like a lot of math, but it's really all doable in semester-1 of an undergrad course, were it focused enough.

View on HN

I think you don’t realise how much math you know. I’ve heard of set theory, but never studied it.

sure, but you dont need to understand actual set theory -- it's just notation for mostly obvious programming stuff,

R means float, Z means int

R^2 is actually notation from linear algebra, but here it means a point is 2 floats with a measure of distance between points etc.

A lot of this could just be given in a "crib sheet" for tech people, and you'd get 80% of it straightaway.

It's years of work to understand this notation as used by the professional domains it was invented by, but it's a couple weeks for most "good, technical, software engineers" -- since they arent really using notation in much more than superficial ways.

Of course there are hard mathematical papers, etc. but they're rare in ML/AI mainstream papers; and not something most would read.

If you need to be able to read, eg., some sort of adv. statistical time series research in economic modelling, you'd already have the background to do that. If all you want is to be able to read 90% of the popular papers, the notation in them is just syntax sugar for things you could state easily in english or python

> sure, but you dont need to understand actual set theory

How would you know? You sound like you understand set theory. Ironically, a body needs a rather extensive grounding in the theory and practice of set theory to look at something and think "I don't need to use the tools of set theory here" with deserved confidence.

A lot of maths is about having a huge repertoire of tricks that you know won't help solve the problem at hand. Saves weeks of fruitless attempts. But I'll bet you didn't notice the things you know not to do. That does matter for interpreting these papers.

> > sure, but you dont need to understand actual set theory

> How would you know? You sound like you understand set theory.

Easy: most mathematicians don't really care about set theory (beyond the fact that it exists and is the standard foundation).

Understanding set theory goes deep. For example: why does the Banach-Tarski paradox exist in ZFC (C: Axiom of Choice (https://en.wikipedia.org/wiki/Axiom_of_choice)), but not in ZFD (D: Axiom of Determinacy (https://en.wikipedia.org/wiki/Axiom_of_determinacy))? Or another one: why does the statement "every field has an algebraic closure" not hold in ZF (but in ZFC)?

Never heard of this stuff despite having studied mathematics? Just like I said: most mathematicians don't really care about set theory.

Not every mathematician needs to understand axiomatic set theory, but probably every working mathematician needs to understand at least basic set operations, simple identites like De Morgan's laws (I don't care if you know the name or not), as well as what cartesian products, relations and functions are.

> every working mathematician needs to understand at least basic set operations, simple identites like De Morgan's laws (I don't care if you know the name or not), as well as what cartesian products, relations and functions are.

This is stuff that you learn in the 5th or 6th grade in school, and is about as far removed from what mathematicians call "set theory" as basic arithmetic operations are from college math courses for soon-to-be mathematicians.

that is a little bit of hyperbole. No school in 5th or 6th grade is giving formal definitions of relations and functions. There is a reason nearly all undergrad math books in analysis, topology, algebra, all devote an entire first chapter to it.

It would be dangerous to teach anyone in ML what a function was, since they might realise, the premise of the entire activity is false.

(ie., that in the vast majority of cases there are no empirical functions to model, no f: Pixel -> Animal)

So better continue the current practice where few could distinguish a relation and a function; and fewer still are aware that they arent approximating empirical functions.

> No school in 5th or 6th grade is giving formal definitions of relations and functions.

I remember that my math teacher pretty surely did.

> There is a reason nearly all undergrad math books in analysis, topology, algebra, all devote an entire first chapter to it.

Indeed there exist multiple good reasons:

- recapitulation

- setting up the notation

- clarifying how the textbook defines the relevant mathematical objects, because the definitions of some concepts might differ depending on the textbook

- making clear what existing standard the textbook expects from the learner

- ...

how do you motivate formal functions and relations to 5th graders, and more importantly why? Relations are important because of the natural partitions of a set they create and the development of group theory. Functions are useful in calculus, but not really the algebraic properties, those are glossed over, i.e kids learning calculus are usually not learning the formalization of functions. That isn't important until analysis or abstract algebra, hence why its included in the textbooks

> how do you motivate formal functions and relations to 5th graders, and more importantly why?

The word "you" in English has two meanings:

1. how would I motive this to 5th graders ("you" as "tu/vous" in French or "du/Sie" in German)?

2. how is this topic motivated in school to 5th graders ("you" as "on" in French or "man" in German)?

For 2: Well, it isn't. The pupils have to accept that in future, they will hopefully get why it is useful. Until then, better learn the material so that you won't fail on the tests.

For 1: If the prophet does not come to the mountain, the mountain must come to the prophet. If you need group theory to motivate functions (as you implicate in your answer), then teach group theory to 5th graders, so that the pupils get the motivation that they desire. If you additionally need to motivate group theory: well, I do know some quite interesting applications of group theory. :-)

Just to make it clear: I do have quite some experience in teaching mathematics, but to highly gifted students.

> Functions are useful in calculus, but not really the algebraic properties, those are glossed over, i.e kids learning calculus are usually not learning the formalization of functions. That isn't important until analysis or abstract algebra, hence why its included in the textbooks

In Germany, there is no distinction made between calculus and analysis. At the university, this subject is taught from beginning on in the abstract way. In school, what you call "calculus" is often taught in a more "hand-waving" way by bad math teachers. Good teachers rather attempt to teach calculus/analysis in the abstract way in school.

fair enough, I was only speaking for the US, which has a slower pace and lack of motivation throughout k-12 + calc series. I still have no idea why they don't teach basic algebra with matrices, so it doesn't seem like a giant bad of tricks, or why linear algebra is so separated from multi-dim calculus, it really makes it more difficult with busy work and you never really comprehend anything.

You may have learned this in 5th or 6th grade, but I certainly didn't.

The point is that any working mathematician is comfortable manipulating sets in algebraic expressions and that's not something you expect from your average high schooler.

I could have added that mathematicians need to know at least about different cardinalities, but I guess strictly speaking you could be working in discrete maths and not care about any of this.

He's saying you don't have to understand set theory to use and understand set notation.

In the same way that you don't have to understand number theory and abstract algebra to use and understand numbers and their basic operations (such as +, -, ·, /).

It would help if instead of saying "set theory" we emphasize the algebra of sets. Not by that name, but this level of teaching sets was common in middle school math late last century, not sure about this century, in the US. And this would be enough for most people trying to read technical documents using set notation.

https://en.wikipedia.org/wiki/Algebra_of_sets

What they’re saying is that you may be falling for the curse of knowledge. From Wikipedia:

> The curse of knowledge is a cognitive bias that occurs when an individual, who is communicating with others, assumes that others have information that is only available to themselves, assuming they all share a background and understanding. This bias is also called by some authors the curse of expertise.

You know which parts of the AI math notation is shallow and which not. Someone else might not have that same knowledge, so they might not know exactly what to put on the crib sheet and what they need to go deeper on.

By the way, if you wanted to make such a crib sheet and publish it online, I think a lot of people would be very grateful!

While I think, that you may be right, that most papers are not about proofs, and that higher math is mostly about proofs, I still think it is math and important math at that. Perhaps it is a link in the chain between mathematics and engineering skills.

The proof centered math is indeed the difficult one. I struggled with a lot at university and was completely unprepared for that finishing school. So I think I have a good idea about the difference in difficulty.

We probably think fluently of these mathematical structures in tangible, real world terms. Still it requires some practice translating to the mathematical abstractions, and still further practice thinking in terms of the abstractions altogether. I don’t know how relevant it is that the textbook content can be covered in a single semester.

>If you could find a book just going through the relevant bits you wouldnt really have to "learn math again"

Hard to find a book or a course doing applied math just for AI and ML. I was interested in learning the math needed to understand quantitative analysis and I was lucky to find some resources doing just that.

This is definitely true. It's largely just notation for communication.

I just paste sections of dense math from AI/ML papers into chatgpt and it explains it. Almost none of it is complicated. It's just really awful notation.

Normalizing flows and neural ODEs are becoming a bigger thing, and they do involve some more heavy maths.

Yes, and if you look at neural operators, then you're in for something that even math majors need to put a bit of effort into.

Oh, I read that paper about a year ago. I feel like it could have been explained better, but the concepts aren't that hard if you've taken numerical methods.

Reply on news.ycombinator.com