Ask HN: How to learn AI from first principles?

The following is not a take that will get you a job or teach you precisely how LLMs work, because you can look that up yourself. However, it may inspire you and you may create something that has a better-than-lottery-ticket chance of being an improvement over the AI status quo:

Without reading about how it's done now, just think about how you think a neural network should function. It ostensibly has input, output, and something in the middle. Maybe its input is a 64x64 pixel handwritten character, and its output is a unicode number. In between the input pixels (a 64x64 array) and the output, are a bunch of neurons. Layers of neurons. That talk to each other and learn or un-learn (are rewarded or punished).

Build that. Build a cube where one side is a pixel grid and the other side delivers a number. Decide how the neurons influence each other and how they train their weights to deliver the result at the other end. However you think it should go. Just raw code it with arrays in whatever dimensions you want and make it work; you can do it in Javascript or BASIC. link them however you want. Don't worry about performance, because you can assume that whatever marginally works can be tested on a massive scale and show "impressive" results.

Interesting idea. I like it.

One can easily build forward pass (forward propagation), for neurons or perceptrons. The learning process itself (which includes gradient descent) is something else entirely, I'd love to learn how to do that, I read a few articles and my stopping point always was multi-variable differential equations, I just couldn't go any further. You could obviously use training functions from existing libraries, but I'm not aware how to do it from scratch...

The question depends what you mean by first principles. Usage of the phrase "first principles" has sprawled into many different things since (I think) Musk first mentioned it as a way to learn. The original, philosophical meaning of first principles meant a fundamental truth which could be used to derive others. Much of the philosophising of thinkers like Aristotle or Descartes was to uncover these truths (eg I think, therefore I am). In physics and other sciences, it means calculations using established laws, rather than approximations or assumptions. Then it got borrowed into certain circles of the tech crowd with the vague meaning of thinking about what's important or true and ignoring the rest. Then it trickled down into the learning/self help world as a hack of some sort to learn. If we take the original meaning of first principles, there aren't a great deal of absolute truths in machine learning. It is a very empirical, approximated and engineering oriented endeavor. Most of the research involves thinking of a new approach, building it and trying it on new datasets.

The other big question is why you want to learn it. If you want to learn ML in itself, than anything including the search algorithms (which used to be considered core to ML a long time ago) you mentioned is part of that. But if you want to learn ML to contribute to modern developments like LLMs, then search algorithms are virtually useless. If you aren't going to be engineering any ML or ML products, what you want is to gain some insight into it's future and the business of it. So learning things like transformer architecture is going to be far more unhelpful than say, reading about the economics of compute clusters.

Given the empirical/engineering quality of current ML, I'd say building it from scratch is really good for getting the handful of possible first principles (the fundamental functions involved, data cleaning, training, etc)

> Usage of the phrase "first principles" has sprawled into many different things since (I think) Musk first mentioned it as a way to learn

In pop culture in 2010+ sure, but he was essentially parroting Feynman IIRC.

"How to learn AI from first principles?"

Start with https://en.wikipedia.org/wiki/Zermelo%E2%80%93Fraenkel_set_t... and eventually you'll get to AI, exercise left to the reader ;)

Ya, the phrase "first principles" is vague...I meant starting from an axiomatic and actionable definition of AI and learning from there. The first chapter of AIMA does a swell job of enumerating different definitions of and then explicitly declaring which one is used and the foundational premises for the concepts and methods to follow. And it doesn't define AI then jump to neural networks, it gradually layers more atomic concepts, like agents (which I know, have been bastardized) and environments, until it gets to machine learning.

> The other big question is why you want to learn it.

Good question. I'm just looking for a wider context to understand contemporary AI. I don't know if this serves any practical purpose but I'm someone who likes to understand the "why" behind everything and starting from "first principles" helps uncover that.

By "first principles" do you mean something long "learn from the ground up" or " from basic building blocks"?

I like learning things starting from small, atomic this, then building up and learning higher layers of abstraction and functionality later. I tend to find hands on totals too "top down" in the sense that they start with all the told in place and then give you a cursory look into what's actually happening.

Personally I feel like most things in the world aren't really that complicated when you understand the building blocks. There are a few core ideas and then a bunch of layers on top to organize and utilize those ideas for different applications. So if I have an interest in something I want to learn from the ground up.

Exactly. Nailed it.

Watch Karpathy's 'Zero to Hero' videos on YouTube.

If you want a historical perspective, which is very worthwhile, start by reading about the mid-century work of McCullough and Pitts, and Minsky, Papert and their colleagues at MIT CSAIL after that.

There will be a dry spell after Minsky and Papert because of their conclusion that the OG neural-network topology that everyone was familiar with, the so-called "perceptron", was a dead end. That conclusion was premature to say the least, but in any event the hardware and training techniques weren't available to support any serious progress.

Adding hidden layers and nonlinear activation functions to the perceptron network seemed promising, in that they worked around some of Minsky's technical objections. The multi-layer perceptron was now a "universal approximator" capable of modeling any linear or nonlinear function. In retrospect that should have been considered a bigger deal than it was, but the MLP was still a PIA to train, and it didn't seem very useful at the scales achievable in hardware at the time. Anything a neural net could do, specialized code could usually do better and cheaper.

Then, in the circa-2010 timeframe, AlexNet dusted off some of the older ideas and used them to win image-recognition benchmark competitions, not by a small margin but by blowing everybody else into the weeds. That brought the multi-layer perceptron back into vogue, and almost everything that has happened since can be traced back to that work.

The Karpathy videos are the best intro to the MLP concept I've run across. Understanding the MLP is the key prereq if you want to understand current-gen AI from first principles.

As a learning exercise, I enjoyed Neural Networks From Scratch: https://nnfs.io/

There's also a world of statistics and machine learning outside of deep learning. I think the best way to get started on that end is an undergrad survey course like CS189: https://people.eecs.berkeley.edu/~jrs/189/

Was not aware of these resources. Thanks for sharing!

It isn't first principles, but I would recommend 3blue1brown's ongoing series about neural networks [0]. I think there's a benefit to seeing the high level overview helps understand the purpose of the pieces as your learning them; it can help with motivation. Or watching overviews like this after the fact it can help bridge connections theory may not elucidate.

[0]: https://www.3blue1brown.com/topics/neural-networks

3blue1brown's content on NN's is awesome -- The explanations are super intuitive. But I'm also looking to understand, as you say, the big picture and understand where NN's fit.

entertainment and not learning

Back in 2018 I did Andrew Ng's course in Machine Learning on Coursera. It was pretty much "from first principles" in that you learned a bit of linear algebra and then you implemented algorithms in Octave, working up to MNIST etc. I felt like I came out of it with a good understanding of the basics, and that ML is maths not magic.

Looks like the course has turned into a multi-course "specialization" and I have no idea if any of it is the aame as the course I did. But it might be a place to start.

I've taken part 1 of 3 in Andrew Ng's machine learning specialization which covers the math for supervised learning, linear regression, etc. As I started part 2 (neural networks), it built off the math from part 1 such as the sigmoid activation function. This is what I think of when the OP refers to learning ML from first principles. I highly recommend Andrew Ng's course and I feel like I need to take it again to really understand those basic building blocks.

1. Brush up on linear algebra (matrix and vector arithmetic, etc) 2. Draw the rest of the fucking owl

can confirm, worked for me.

i am now applying to current yc batch.

Hell, all you need to do for YC is fork continue.dev and name your company after a fruit.

Hi, it depends on what you mean by "first principles".

If you don't have a solid background in math, then that's what you should improve upon (calculus, linear algebra, discrete math, probability theory, information theory). Some of the books you mention do cover this at the beginning, but most people take separate courses on these topics at University, with lots of homework, etc.

Also, the first book on your list is the classic textbook by Norvig, but I don't think it's actually very good. I remember reading it in my college AI course 25 years ago and it was painful back then (anybody remember "wumpus"?). It's a big book that covers too much, it's like printing out a lot of Wikipedia pages. You're better off finding books with smaller scope that focus on something you actually care about / is relevant to the way the field has developed.

AIMA is wide-ranging, a lot of which is not "must-have", but "nice-to-have" knowledge. But I do like its breadth-over-depth approach to get a full scope of the AI landscape.

https://a16z.com/ai-canon/

I prefer the a16z AI canon for this purpose. It’s useful and historical. It’s structured to begin with no prerequisites and work up to cutting edge research papers. And best of all it’s free and open source.

For deep learning:

1. Linear algebra. Be comfortable with vector transformations in the vector space. This is the framework to understand how data is represented and what is going on inside the model.

2. Calculus. Specifically derivatives, up to partial derivatives and the chain rule. This is needed to later understand backpropagation, the learning. It's fine to skip integrals.

3. Vanilla neural network. Study how a simple feed forward and fully connected neural network works, in detail. Every single bit about it.

I wouldn't worry or plan anything ahead until having those. After number 3 you'll have different branches to follow and will be better equipped to pick a path.

Do you have specific courses to recommend? Linear Algebra is probably a good one to be learning from Gilbert Strang.

I prefer learning from textbooks than online courses, here are my picks (in the order of my first post):

1. Sorry, I do not have a suggestion here. I'm Brazilian and I happened to find an amazing PDF of an obscure book (by a not well known professor). In Portuguese though.

2. "Calculus Made Easy" is my book of choice. Don't dismiss it because of the 'cheap' title, it's a very good book.

3. It's super important to get the intuitions first, and not just the math. So I mostly used two books, both heavily illustrated: "Deep Learning - A Visual Approach" and "Deep Learning Illustrated". Those are way better as an introduction to the field than the 'big names' (Goodfellow/Bengio, Bishop, etc).

Normally if I'm on youtube, it means I'm struggling with something. The statquest channel is amazing, it's the most 'baby steps' explanations I've ever seen. If I find a video on what I want to understand there, I don't waste my time trying other channels anymore.

I’m starting down a similar path with these two books

How AI Works - https://nostarch.com/how-ai-works

+

Why Machines Learn: The Elegant Math Behind Modern AI - https://www.penguinrandomhouse.com/books/677608/why-machines...

Thanks for the recommendations. What's your impression of these texts so far?

I’m enjoying How AI Works but I have not made it to the other yet.

A loved AI: A Modern Approach—still the best all around textbook on AI imho. I'd only add Susanna Epp's Discrete Mathematics with Applications, and I'd just focus on methodically and thoroughly working through those two, personally. First, Discrete Mathematics, then AI: A Modern Approach. This is exactly what I did and it was a great experience, super helpful.

Will check out Discrete Mathematics, thanks!

I've been a fan of The Little Learner for the first principles side of things. It builds up the theory from almost nothing, step by step. It's got a conversational style that may turn some off, but I quite enjoyed it.

https://www.thelittlelearner.com/

seconded

Step 1, abandon the concept of "first principles." First principles don't exist for most areas of study outside of pure mathematics, and for newer fields like AI they haven't been established yet so you'd just be hamstringing yourself.

Step 2: The steps above are a good plan for learning about traditional AI, and the traditional approaches, which were based on an attempt to model human thought processes. Machine learning was what the industry turned to in the early 2000s because we didn't have the hardware capabilities then to meaningfully model neural networks. We do now, but machine learning has taken over so there's very little research into modeling neural networks...about the same as there was when I was an undergrad.

The question is rather a bit flawed without knowing your background, as to determine what to suggest. That being said, I would argue a lot of the novelty and recent advances in AI/ML are not yet documented in books, but rather in dense scientific papers.

Artificial Intelligence by Poole and Macworth was a pretty solid,and free, book when I read it for a class.

https://www.artint.info/

Great recommendation. Based on the ToC, its similar to AIMA.

What these guys are on about: https://arxiv.org/abs/2301.10743

Just do Andrew NG's course.

It will show you the maths, you'll build simple neural nets (from maths!) that can read digits and scale it from there.

While doing it, you may struggle with some of the maths, just take a deep breath and invest sometime to fill the gaps you need and continue.

Learn about CNNs and everything else step-by-step. It's awesome.

Andrew Ng's course is great for the learning NN's from scratch, but not understanding how NN's fit in the broader discipline of AI.

https://udlbook.github.io/udlbook/

I recommend the following video lectures from a course: https://www.youtube.com/@deeplearningsystemscourse1116/

It covers ground I don't know where else to find on getting from theory to practice.

Kind thanks for this...

When a notion is disturbing before dismissal I must know why (MY why) 90% boils down to fear (mine) per lacking information hence failure to understand.

The discussion & refences found aided much for my understanding

I really enjoyed the concepts in "Artificial Intelligence, a Modern Approach" which really grounds a first-principles foundation of automated reasoning. Warning: The first 80% of the book doesn't have any sexy new deep learning approaches, but I still think it is very valuable to see the history.

+1, I'm a few chapters in and its highly instructive. Gives me a deeper appreciation for the modern deep learning regime. Also, as we enter the agent supercycle, I think many of the basic algorithms for search, planning, etc. will make comeback a in a huge way.

In the coursera course accompanying the statlearning book he starts with introducing a nearest neighbour algorithm and from that he develops a linear regression.

I think that is "first principals of AI". Like, what does it even mean when we ask an algorithm to "learn" from data?

I think Ian Goodfellow's "The Deep Learning" textbook fits well to the list.

https://www.deeplearningbook.org/

This article takes a journey from first principles: https://harishnarayanan.org/writing/artistic-style-transfer/

Does anyone know how to purchase the PDF version of the book: "Artificial Intelligence, a Modern Approach"?

No, but there is a PDF version, ahem, floating around on Reddit.

I want to study game AI but I don’t think I have found much material yet. I guess it is too niche now

https://aima.cs.berkeley.edu/ has some good material on game AI. Its surface level but it does discuss SOTA methods like AlphaZero.

Heard nothing but good things about Andrej Karpathy's series on Youtube

Yeah but once micrograd is built, he doesn't expand on that and jumps to bigrams etc.

Going to plug the thoroughly excellent fast.ai courses here

IMO learning AI from first principles means a Math/CS Master from a reputable CS university, preferably top 20.

Ask them?