Hacker News Reader

Comment by whartung | Hacker News Reader

Can I use any of the information in this book to learn about reinforcement learning?

My goal is to have something learn to land, like a lunar lander. Simple, start at 100 feet, thrust in one direction, keep trying until you stop making craters.

Then start adding variables, such as now it's moving horizontally, adding a horizontal thruster.

next, remove the horizontal thruster and let the lander pivot.

Etc.

I just have no idea how to start with this, but this seems "mainstream" ML, curious if this book would help with that.

View on HN

I enjoyed "Grokking Deep Reinforcement Learning"[0]. It doesn't include anything about transformers though. Also, see Python's gymnasium[1] library for a lunar lander environment, it's the one I focused on most while I was learning and I've solved it a few different ways now. You can also look at my own notebook I used when implementing Soft Actor Critic with PyTorch not too long ago[2], it's not great for teaching, but maybe you can get something out of it.

[0]: https://www.manning.com/books/grokking-deep-reinforcement-le... [1]: https://gymnasium.farama.org/environments/box2d/ [2]: https://github.com/DevJac/learn-pytorch/blob/main/SAC.ipynb

Reinforcement learning is an entirely separate area of research from LLMs and, while often seen as part of ML (Tom Mitchell's classic Machine Learning has a great section on Q learning, even if it feels a bit dated in other areas) it has little to do with contemporary ML work. Even with things like AlphaGo, what you find is basically work in using deep neural networks as an input into classic RL techniques.

Sutton and Barto's Reinforcement Learning: An Introduction is widely considered a the definitive intro to the topic.

Sorry, in that case I would rather recommend a dedicated RL book. The RL part in LLMs will be very specific to LLMs, and I will only cover what's absolutely relevant in terms of background info. I do have a longish intro chapter on RL in my other general ML/DL book (https://github.com/rasbt/machine-learning-book/tree/main/ch1...) but like others said, I would recommend a dedicated RL book in your case.

Try OpenAI's spinning up: https://spinningup.openai.com/en/latest/

This is a good and short introduction to RL. The density of the information in Spinning Up was just right for me and I think I've referred to it more often than any other resource when actually implementing my own RL algorithms (PPO and SAC).

If I had to recommend a curriculum to a friend I would say:

(1) Spend a few hours on Spinning Up.

(2) If the mathematical notation is intimidating, read Grokking Deep Reinforcement Learning (from Manning), which is slower paced and spends a lot of time explaining the notation itself, rather than just assuming the mathematical notation is self-explanatory as is so often the case. This book has good theoretical explanations and will get you some running code.

(3) Spend a few hours with Spinning Up again. By this point you should be a little comfortable with a few different RL algorithms.

(4) Read Sutton's book, which is "the bible" of reinforcement learning. It's quite approachable, but it would be a bit dry and abstract without some hands-on experience with RL I think.

That's exactly what the Q-learning lab in this course does:

https://www.ida.liu.se/~TDDC17/info/labs/rl.en.shtml

This book seems to focus on large language models, for which RLHF is sometimes a useful addition.

To learn more about RL, most people would advise the Sutton and Barto book, available at: http://incompleteideas.net/book/the-book-2nd.html

I would recommend this as a second book after reading a "cookbook" style book that is more focused on getting real code working. After some hands-on experience with RL (whether you succeed or fail), Sutton's book will be a lot more interesting and approachable.

Reply on news.ycombinator.com