Dopamine framework: Fast prototyping of reinforcement learning algorithms (2018) | Hacker News Reader

Dopamine framework: Fast prototyping of reinforcement learning algorithms (2018)

ai.googleblog.com

81 pointsrbanffy15 comments

I would really like the opportunity to get into RL. At work my team uses GANs and RNN/LSTMs almost exclusively. Any suggestions for specific RL online classes at Coursera, eDX, or Stanford?

I especially like that this framework is set up for experimenting with four popular models and the means to measure how well your models are doing.

I came across this a few days ago: For Reinforcement Learning specifically, the standard text is Reinforcement Learning: An Introduction[1]. Dave's UCL Course on RL[2] is great too (playlist of all lectures)[3].

Source: Julian Schrittwieser works on Deepmind at Google http://www.furidamu.org/

[1]http://incompleteideas.net/book/the-book-2nd.html

[2]http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html

[3]https://www.youtube.com/watch?v=2pWv7GOvuf0&list=PL7-jPKtc4r...

Thanks for this list. I’d like to get up to speed on RL and see how we can apply it for path planning and control of our gliding parachute UAVs.

Ooh! I want one of those for high altitude balloon payload recovery. Got any small ones or hobby grade projects doing this that you’re aware of?

No hobby ones I know of, although I haven't looked into the hobby space too much. All I can say about smaller systems is give our BD guys a call... =D

thanks!!

Beyond those, my favorite resources are:

- "An Introduction to Deep Reinforcement Learning" by Vincent François-Lavet et al (https://arxiv.org/pdf/1811.12560.pdf)

- "A (Long) Peek into Reinforcement Learning" by Lilian Weng (https://lilianweng.github.io/lil-log/2018/02/19/a-long-peek-...)

- "Deep Reinforcement Learning: Pong from Pixels" from Andrej Karpathy (https://karpathy.github.io/2016/05/31/rl/)

Those are the basics. Some more resources listed on this post: https://news.ycombinator.com/item?id=18219620

If you are not an expert I can recomment David Silver's lectures on the topic. There is a playlist on youtube (https://www.youtube.com/playlist?list=PLzuuYNsE1EZAXYR4FJ75j...) and from what I have seen it is quite well explained.

This goes over the theory behind RL mostly, so if you are looking for practical implementations you might want to find other resources.

thanks, I will look at Silver’s lectures. I have not finished reading Sutton and Barto’s RL book yet, so I am still a beginner.

RL is really really brittle, i.e. one simple bug that wouldn't matter in GAN/LSTM (such as e.g. adding one more or less number than needed) would cause your RL algorithm not to converge and you'd spend days-weeks debugging it - it requires extreme discipline verifying every single step of your algorithm. Often you don't know if your model didn't converge because it is not strong enough or if there was some tiny little error/wrong assumption leading to divergence.

Yes, some types of NNs are resilient to errors in implementation.

I hired David Rumelhart as a consultant in the 1980s (when he was still at UCSD) when I was writing SAIC’s neural network product for Windows. He warned me that he was used to seeing projects with big implementation errors that still worked very well.

I do find GANs brittle. I use them for generating numeric/categorical spreadsheet data, not for images.

> He warned me that he was used to seeing projects with big implementation errors that still worked very well.

For any sufficiently complex function, if you tune the constants enough, the wrong function will sometimes approximate the results the correct one would give if it were set up with the correct constants.

> I do find GANs brittle. I use them for generating numeric/categorical spreadsheet data, not for images

I’m interested to hear more about this. Care to elaborate?

May I ask out of curiosity what commercial application are you using GANs for if any?

CS 294 is a great deep reinforcement learning course that I highly recommend [1]. Sergey Levine is god tier RL pretty much.

[1] http://rail.eecs.berkeley.edu/deeprlcourse/

Reply on news.ycombinator.com