How does this compare to the karpathy video [0]? I'm trying to get into LLMs and am trying to figure out what the best resource to get that level of understanding would be.
- it implements a real word-level LLM instead of a character-level LLM
- after pretraining also shows how to load pretrained weights
- instruction-finetune that LLM after pretraining
- code the alignment process for the instruction-finetuned LLM
- also show how to finetune the LLM for classification tasks
- the book it overall has a lots of figures. For Chapter 3, there are 26 figures alone :)
The video looks awesome though. I think it's probably a great complementary resource to get a good solid intro because it's just 2 hours. I think reading the book will probably be more like 10 times that time investment.
I've watched it many times to understand well most of it.
And obviously you must already know pytorch really well, including the matrix multiplication, backpropagation etc. He speaks very fast too...
In my opinion he covers everything needed to understand his lectures. Even broadcasting and multidimensional indexing with numpy.
Also in the first lecture you will implement your own python class for building expressions including backprop with an API modeled after PyTorch.
IMHO it is the second lecture I can recommend without hesitation. The other is Gilbert Strang on linear algebra.
There is a lot to learn, but I think he touches on all of the highlights which would give the viewer the tools to have a better understanding if they want to explore the topic in depth.
Plus, I like that the videos are not overly polished. He occasionally makes a minor blunder, which really humanizes the experience.
Anyway those videos are quite advanced. Surely not for beginners.