Hacker News Reader

Comment by rfrey | Hacker News Reader

I love textbooks and spend more of my childrens' inheritance on them than I should.

But what MOOCs give me is the exercises. I often think I understand a problem, but it's only after getting 2.1/10 on a Coursera quiz that I realize I've missed a key step or concept.

Many textbooks have exercises but few have solutions. I've been working through Barto and Sutton's Reinforcement Learning for example (again), and although I do the exercises and programming questions, I never know if I've gotten it right. My experience with MOOCs shows I probably haven't in a large number of cases.

The best of both worlds is when I can follow a MOOC with the textbook to gain more depth, for example with the PGM course on Coursera.

View on HN

Textbooks also have "examples" which are exercises with worked answers. If you cover up the solution and try to work through it on your own first, a worked example is very instructive.

It's just as instructive (and usually pretty embarrassing) to go back and work through the examples a day or two after you first read them. And then a day or two after that :)

Yes indeed. It's the exercises and feedback that help a lot.

I believe that the optimized future is MOOC Problems to start, Texts and Recorded lectures to help where you struggle, humans to help when the texts and recording miss your mistakes.

Incidentally, I liked Barto and Sutton's book, but it feels a bit dated. Does anyone have another reinforcement learning book to recommend?

I chatted with Rich Sutton recently and he said he was writing and update - not sure how far along in the process he is though.

Also Csaba Szepesvári (a colleague of Rich's at the U of A) has a free RL book you can download. http://www.ualberta.ca/~szepesva/RLBook.html

Csaba's book is the most up-to-date on RL I know of. Sutton and Barto is very old by now. For the POMDP side of things there are no recent books I know of, but http://www.cs.mcgill.ca/~jpineau/files/sross-jair08.pdf is a recent enough survey.

A related book is "Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems" which you can find at http://www.princeton.edu/~sbubeck/index.html

The bandit problem is very strongly related to the reinforcement learning problem, so you'll get some mileage out of studying bandits. Be aware this area is very maths heavy, which is good or bad depending on your background. If you like you like this stuff, also checkout "Prediction, Learning, and Games" which deals more with the "adversarial" setup.

Agree with others that Barto & Sutton is dated, but it's not obsolete. I was chatting a couple weeks ago with one of Sutton's former students, who has since Made Good in both academia and industry working on AI, and he said that the B&S book is all you need to be able to read the literature, so it's still a good place to start.

Apparently Sutton was going to do a second book, but has since retreated to a second edition. I'll take it though. :)

The Barto and Sutton book is available online too, http://webdocs.cs.ualberta.ca/~sutton/book/the-book.html although the printed copy is a high-quality publication so maybe worth it.

Agreed, it's also a bit topical.

Check out Wiering + van Otterlo's "Reinforcement Learning: State-of-the-art." Covers many new techniques--I used it as a reference for a project earlier this year:

http://www.springer.com/engineering/computational+intelligen...

Wow, $280. I wonder if my public library will spring for it...

If you're a student you can likely get a PDF for $0 through your library. But...yeah. Steep price.

Reply on news.ycombinator.com