The Mathematics of Artificial Intelligence (2022) | Hacker News Reader

The Mathematics of Artificial Intelligence (2022)

arxiv.org

181 pointsofou59 comments

After a cursory glance, my feeling is that "The Mathematics of Neuronal Networks" would be a better title?

In recent years, the term "Artificial Intelligence" is often used instead of "Neuronal Networks".

I wouldn't be surprised if this will change again. If there is evidence that it will not and Neuronal Networks are for some reason the optimal medium for intelligence, I would love to read about it.

Came here to say the same thing. This is not a paper on mathematical artificial intelligence, at last as I understand it, not in the way Goertzel or Hutter write about it. I was actually thinking this would be something like Hutter's universal artificial intelligence.

This is a very good point. To anyone interested in what Artificial Intelligence looked like before neural networks, the "Artificial Intelligence - A Modern Approach" by Stuart Russell and Peter Norvig is a good book on the subject.

Agreed. I was teaching from it (AI 101, search/agents, no ML) and it's possibly the best textbook I have encountered. It's structured very well.

As an aside, I will forever remember the rough geography of Romania. I still plan to do an A* route visit of the country some day :D

I like to explain that AI > ML > NN.

Now, NNs are the ones getting results at computer vision and natural language, and more. I think most people would say that other ML approaches are computational statistics. The goalpost for AI keeps moving.

If you are truly interested in the math of AI I think PAC Bayes learning is more appropriate and your book is Understanding Machine Learning [1] (not an easy read). A more gentle intro would be Learning From Data [2]. If someone recommends a book/paper it would be awesome, I'm always on the look.

[1] https://www.cs.huji.ac.il/w~shais/UnderstandingMachineLearni... [2] https://work.caltech.edu/telecourse

Seconding this. "Learning From Data" is one of the best intros to machine learning theory I've seen through the years!

NN > ML. The proof is that nematode or fly, I dont remember, with its simple NN fully mapped, and still remaining a mystery how it works. ML, which is just a matrix multiplication at its core, is a laughably simplistic model of NN.

GP means artificial neural network, not an actual nervous system.

Machine learning contains ANNs as a sub-discipline. Other non-ANN topics in ML include ensembled trees, Gaussian processes, and sampling theory.

linear-algebraic matrix multiplication complicated by non-linear threshold discontinuities evaluators, no?

I think one can't compare deep learning to machine learning since they have different purposes. While there have been great strides in the interpretability of NNs, the analytic models of classical machine learning are favored from a computational and interpretability perspective. Conversely, if the relationships in our data are constantly evolving (e.g. the change in the stochastic process followed by a time-series from one interval to another), then NNs are more appropriate.

Although it tends towards Deep Learning as opposed to AI, I highly recommend Bishop's Pattern Recognition and Machine Learning. It not only provides a solid Bayesian perspective, but also comments on the subtleties of applying theory. For the latter, its discussion of overparameterization in (I think?) the first chapter comes to mind.

[1] - https://www.microsoft.com/en-us/research/people/cmbishop/prm...

> Neuronal Networks are for some reason the optimal medium for intelligence

No, that's not the same 'intelligence' ("General Intelligence") as the "I" side in Artificial Intelligence.

The term 'intelligence' applied to Artificial Neural Networks makes sense, as such: to reach a procedural solution it takes an engineer; the engineer is said to have reached the solution because "intelligent"; ANNs are (semi-)automated builders of function approximators; ANNs are said "intelligent" because they "reach solutions" (like the engineer would have done - "approximator":"engineer"="Artificial":"Natural" Intelligence). And of course they are not "intelligent", while they are in some sense. It's just an expression, it's rhetoric (it requires considerate interpretation).

That some sort of "intelligence" is achievable in other ways, or that ANNs spawn as an idea from anatomical considerations of naturally intelligent entities, or that ANNs could help in modelling general intelligence (etc.), is tangential.

ANNs dont have the key component of intelligence - ability to learn. If our brains worked like ANNs, they'd be wired once at the brains factory, they'd produce the same output given the same input, and learning anything new would need a complete rewiring at the factory, possibly erasing other learnings (often unpredictably).

Even if ANNs could update themselves on the fly, in a reasonably incremental manner, they'd just imitate insect-level reactions. Intelligence would need those ANNs to have a virtual reality, run thought experiments there and learn from that.

Well this does not mean that some mechanisms similar to those we are studying cannot be part of the original or that they cannot be part of a General Intelligence system. Even if that were simply "adjusting a structure to function and information retention". The very "compositional" nature of the perceived that makes Convolutional ANNs work was noted after biomimicry.

This, of course, as an aside.

You're asking for online learning. This is certainly possible with modern techniques - define an unsupervised loss and update the weights with gradients as new data is encountered. Federated learning also works along these lines.

'Raw' online learning is unpopular because you have no garauntee that the network won't do something funky in the field.

That said, I think there are production systems in the world which learn on a day by day basis. Eg, take all of the logs from the last day and use them to update the production model for tomorrow. Then there's enough data that you don't risk a bad step. Think of it as learning while dreaming...

We usually say "neural networks" and not "neuronal networks"

I think in Greek 'neura' and 'neuron' are equivalent. The perception that '-on' indicate the single or ideal entity ("ion, electron, muon...") comes from Whewell and Faraday ('-on' ex "ienai": the "goer" particles) and fully postdates the use of 'neuron'.

Edit: also see the comment from member martopix, nearby ( https://news.ycombinator.com/item?id=30988150 )

You have the correct etymology, but the convention is for "Neural Network" in ML and "Neuronal Network" in Neuroscience for we laymen.

As noted in the reference to the post from martopix :)

Because AI basically "stole" the term neural networks from neuroscience, neuroscientists now often speak of "neuronal networks" while referring to biological ones, to avoid confusion. So I would recommend to not use "neuronal networks" in this context.

I’m on Chapter 4 of the Mathematics for Machine Learning. Enjoying it so far:

https://mml-book.github.io/

After learning the math, which machine learning books come next?

If you are looking to have a mathematical understanding of the field then I really enjoyed "Understanding Machine Learning: From Theory to Algorithms" by Shalev-Shwartz and Ben-David.

When you compare it to Edward Bender's old book[1], the "math of AI" seems to have become a very narrow subject.

[1] https://www.amazon.com/Mathematical-Methods-Artificial-Intel...

Does anybody know of textbooks, or articles, material etc. that focus on the /consequences/ of the mathematics of ANNs? For example: why in this problem using more layers is more or less efficient than using bigger layers, or branching evolutions of the outcomes as opposed to intensive computation of a single flow etc.

A reasoned summary of the tricks, in a way ("this works because of that").

Agree

Some of the questions of depth vs breadth are explained intuitively in this Bengio Lecun paper from 2007.

http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf

Check this out:

https://ai.facebook.com/blog/advancing-ai-theory-with-a-firs...

HN discussion:

https://news.ycombinator.com/item?id=27559017

My own review comment:

https://news.ycombinator.com/item?id=27564506

Edit: Corrected the third link. Thanks mdp2021 for notifying me of the copy-paste error.

Thank you alok-g,

I would just correct the links:

-- article: https://arxiv.org/abs/2106.10165

-- book: https://arxiv.org/pdf/2106.10165

-- your summary: https://news.ycombinator.com/item?id=27564506 (the link to the post is that "under" the date. You posted two identical links here)

Thanks, this is great.

Poorly thought out morning spitball coming. One of the reasons approximation theorems are so unsatisfying is that they are always of the form "for function class X there exists an architecture A of complexity O(N) such that blah". And then this is compared favorably with some other function class whose dimension is O(N). But there's something tricky about this: you leave the architecture unspecified. You are comparing a single space of functions with an enormous number of spaces, one for each architecture with the specified complexity, and then saying "well if I pick the right architecture I win". Doesn't seem like a fair comparison.

more spitball -- the term "AI" changes the context from machine-side to application-side. "AI" implies to the listener that the machine is doing something that only thinking humans could do.. emphasis on the humans, while sets of math that seek minimized error, emphasize the behavior of the functions.

Along the same lines, here is a course on the "Mathematical Foundations of Machine Learning" [1] (from GT, but other institutions also having offerings) and is a section in the paper in this post (so deeper dive possible):

[1.] https://jrom.ece.gatech.edu/mfml-f20-notes/

Shouldn't "the mathematics of artificial intelligence" be subsumed by computational statistics? It may be that only very general and weak statements are possible about "deep" NN's, but then that's merely a fact about the territory we're dealing with. Why should this be regarded as a "new" field?

Link to vanity version (a web version of the pdf): https://www.arxiv-vanity.com/papers/2203.08890/

who runs this site? where are the logs stored?

Let L(uy,y) = fy denote a parametric partial diﬀerential equation with y being a parameter from a high-dimensional parameter space Y ⊆ Rp and uy the associated solution in a Hilbert space H. After a high-ﬁdelity discretization...

These AI folks are clearly very clever, but they don't actually believe that's got anything to do with how human thinking works, right?

Research on natural intelligence or general intelligence presents itself as such.

AI leaders occasionally seem to pontificate on AGI, yet it's difficult for a layperson like me to see how their expertise (as seen in publications like this) translates.

Whenever I see "Hilbert space" remarks in a paper that doesn't concern abstract math, I roll my eyes and imagine a chef who starts his youtube video with "Today we are going to cook a salad. For better precision, we are going to use knifes of triangular shape, three dimensional plates of circular shape, and the 100% dihydrogen of oxyde solution." All this only to obfuscate the fact that the entire recipe is just chopping a few carrots.

Try doing quantum mechanics, even at beginners level, without a proper knowledge of Hilbert spaces.

the subclass of "Reproducing kernel Hilbert spaces" form the basis for a large class of ML algorithms (kernel methods)

It's literally called "the mathematics of ARTIFICIAL intelligence"

The important word is intelligence.

Is it correct to use "we" if there is only one author?

A form of "plural majestatis" which paradoxically implies that the individual author has only relative importance

(«paradoxically» as the majestas itself is founded on representing something higher, not of some high status of the individual which happens to be endowed - it may sound like boasting but on the contrary it would be a downplay of the individual).

In other contexts, that "we" can have even more foundations: "I could never have done this without the work of others", "Not just me but all those who think alike" etc.

It's standard practice for mathematics papers. Some people like to view it as "we, the author and the reader".

It's the "royal 'we'"

Using the "royal we" is standard in mathematics. If you ever see someone do this in ordinary writing you can bump up your odds that they're a mathematician. Another giveaway is using "modulo" to mean "except for".

me and cockroaches in my head

Pity that it uses Artificial Intelligence instead of the more correct and precise term Machine Learning, especially since Mathemticians are supposed to be more precise.

It is mathematically correct: what concerns ML concerns AI.

The interest of the author is explicitly ("precisely") on AI, where, statedly, «the current “workhorse” of artificial intelligence [are] namely deep neural networks».

Has anyone followed Smolensky’s Harmonium approach? I think it is so cool that “goodness of fit” was based on “harmony”…

Have a search under the alternative name Restricted Boltzmann Machine if you haven't. Several interesting developments since the 80s.

Oh yeah well aware that it was the first restricted Boltzmann Machine. It minimizes energy, but that’s just harmony with a sign change. When Smolensky Hinton and Rummelhart collaborated, they decided to call it “goodness of fit.” So neural networks still optimize for harmony by another name. I think that’s so cool. The original Smolensky paper is worth reading. Good ideas.

I finished my studies in CS less than a year ago. Nothing about this paper is new, but it gives a quick overview.

No need to be snarky. It could be new for someone just getting into the field.

The poster probably only meant to convey that the article is compilative (a summary, as opposed to new research) for those who are up to date.

Is "Nothing about this paper is new" an appropriate response when the abstract explicitly mentions that it's a survey paper? Maybe I'm just sensitive, but to me that sounds dismissive rather than purely informative.

Well, the poster may have just rephrased the formula («survey article»), or may have noted it for the benefit of the skimmer, or just used it for the conclusion out of their judgement "for an overview it does its job". «sensitive ... sounds»: the author is apparently a native German speaker and not only different minds and cultures weigh expressions differently: there could even be a linguistic bias. We are mandated and obliged to «Assume good faith».

> Nothing about this paper is new, but it gives a quick overview.

Just as the paper itself clearly states. So your comment comes across as rather rude.

Reply on news.ycombinator.com