Reinforcement Learning: An Introduction (2018) | Hacker News Reader

Reinforcement Learning: An Introduction (2018)

incompleteideas.net

120 pointsibobev23 comments

The field of reinforcement learning is huge. There are many different algorithms with catchy names, which are mostly variations of two basic algorithms. However, each variation can easily keep you busy for a PhD or two.

The book referenced here (the "Sutton and Barto book") covers the basics. If you like more, or more recent texts, consider these (mostly available for free):

Multi-Agent Reinforcement Learning: Foundations and Modern Approaches, 2024, https://www.marl-book.com/

Distributional Reinforcement Learning, 2023, https://www.distributional-rl.org/

Deep Reinforcement Learning, 2022, https://deep-reinforcement-learning.net/

Related. Others?

Reinforcement Learning: An Introduction (2018) [pdf] - https://news.ycombinator.com/item?id=19191746 - Feb 2019 (23 comments)

Reinforcement Learning: An Introduction, Second Edition - https://news.ycombinator.com/item?id=18547998 - Nov 2018 (6 comments)

New Draft of “Reinforcement Learning: An Introduction, Second Edition” - https://news.ycombinator.com/item?id=12568414 - Sept 2016 (33 comments)

Reinforcement Learning: An Introduction - https://news.ycombinator.com/item?id=1083662 - Jan 2010 (4 comments)

This book seems to be very theoretical. Can someone recoment more practical books with code samples using some modern ML framework, probably something like _Hands-On Machine Learning_ by Geron Aurelien

https://course.fast.ai/

"You’ll see that fast.ai’s way of teaching is very different to what you might be used to, if you did a technical degree at university. Nearly all technical subjects at university are taught “bottom up”: start with basic foundations, and gradually work up to complete useful solutions to real world problems. But we go “top down”: start with complete useful solutions to real world problems, and gradually work down to the basic foundations. Education experts recommend this approach for more effective learning."

I don't think this discusses reinforcement learning (not sure about the comment you responded to). To my recollection, the first course goes through basic architectures and concepts, and the second discusses autoenvoders.

Yeah, the Fast.ai course does not discuss RL. The first part focuses on DL and the second part focuses on Diffusion based models

To be honest I think this may be actually an advantage: it explains concepts which otherwise are just weird parameters in code. Since it is pretty lengthy I would actually recommend to read the chapters relevant to a specific method you are interested in (maybe going a backwards to build the right context).

If you I'd combine it with e.g. https://spinningup.openai.com/en/latest/ or doing some toy projects with https://stable-baselines3.readthedocs.io/en/master/ it would probably render the most value.

I second this. When I started learning ML, rather than getting into coding I first watched Yaser Abu-Mostafa's Caltech lectures on YouTube diligently. I used his book as a companion. Next I watched Robert Tibshirani's lectures. I was interested in learning the fundamentals, which helped me a lot when I started coding.

"Grokking Deep Reinforcement Learning" by Miguel Morales and "Deep Reinforcement Learning in Action" by Alexander Zai and Brandon Brown both look promising, though the code might be outdated. Looks like they use the OpenAI Gym environment, which has since been forked and maintained as Gymnasium.

I made a small project where I show how to train a Neural Net with Q-Learning for a simple temperature control problem. Still a hard read for someone with no background in math/control theory probably, but the code is actually much easier than the text^^

https://github.com/the5avage/Q-Learning/

I did the Reinforcement learning specialization on Coursera (https://www.coursera.org/specializations/reinforcement-learn...), which used this book. The specialization was great, but I would say that I had a lot of trouble following this book. I would find that there was a concept I didn't quite get, and then other things would build on top of that concept. I ended up feeling like I didn't quite grasp everything, even though I read most of the chapters more than once.

I do feel like there are a lot of things that aren't fully explained in this book, and maybe it is expected that the reader has some prior knowledge that I didn't have. For example, they never really explain the basic notation for backup diagrams. They just show one and mention that they are a thing. The mathematical notation they use also isn't really explained - they just start showing formulas of increasing complexity. It's possible to look these things up elsewhere, but they probably could have spent just a little bit more time explaining some of the basics and made it easier to follow.

What made the specialization great? Did they explain things well where the book didn’t? Are the courses self-contained? What are the prerequisites?

The specialization did a good job of explaining all of the basic concepts of RL in a fairly understandable way. The instructors don't assume a huge amount of prior knowledge, and make it pretty accessible. They split reasonably between theory and applications. I would say that you need some background in math and ML.

The courses do build upon each other. You could do just the first course and get a good overview of RL without diving into the details, but I don't think you could start midway through the specialization.

I would also suggest looking at the Hugging Face Deep Reinforcement Learning course. That one is very different (focused more on application than diving deep into the theory), but it's taught by a non-academic and really tries to explain the concepts in a way that is approachable to most programmers.

Does anyone have any feedback on this Udacity courses:

Deep Reinforcement Learning

Nanodegree Program

https://www.udacity.com/course/deep-reinforcement-learning-n...

There's lecture notes on this book available from David Silver at UCL: https://www.davidsilver.uk/teaching/

Thanks for sharing this. David Silver's YouTube lectures on RL are great too

Do we have any real world applications where the policy satisfies strictly some constraints (think physics: conservation of mass etc)? There is research in the field but not sure if anything is in production.

This is what I'd like to know as well.

Interesting to read the last section of the last chapter (17.6 Reinforcement Learning and the Future of Artificial Intelligence) given that the book is from 2020 and ChatGPT (in which RL plays a key role) was published in 2022.

i pretty much have read this book. pretty boring tbh but still good. i would recommend doing hands on implementations with the cartpole environment

What are the math prerequisites for this book? What other reinforcement learning books or courses are recommended for beginners?

This is THE book. It is suitable for beginners with basic math background equivalent to CS undergrad I’d say

Basic probability/stats and some basic fundamentals around dynamic programming/recursion would be very helpful.

The big problem I found with this field is that the core ideas are very subtly built on top of each other. Without a proper teacher or an environment to study, self-study is much much harder.

(Past chapter 5, it should be a breeze as the foundation would have been strongly set)

Reply on news.ycombinator.com