Hacker News Reader

Comment by rrherr | Hacker News Reader

Best paragraph:

"Cracking AGI is a very long-term goal. The most relevant field of research is considered by many to be reinforcement learning: the study of teaching computers how to beat Atari. Formally, reinforcement learning is the study of problems that require sequences of actions that result in a reward/loss, and not knowing how much each action contributes to the outcome. Hundreds of the world’s brightest minds, with the most elite credentials, are working on this Atari problem."

View on HN

Calling it an "Atari problem" sounds quite disparaging and misses the point. It's like calling a convolutional network doing the ImageNet task a "Doggy-detection" problem. That may be the original development problem, but the final product still helps detect cancer in CT scan images... Same goes for advances in reinforcement learning made on atari games.

Perhaps, but the jury is still very much out. The vast majority of RL applications are game playing. Very few examples of valuable applications to society or the economy.

There's also plenty of evidence already that RL isn't really the right way to tackle the credit problem. E.g random search is only 10x slower.

Also used for robotics: https://blog.openai.com/roboschool/

and other control problems: https://vmayoral.github.io/robots,/ai,/deep/learning,/rl,/re...

I think not so long ago supervised learning felt similarly 'toy', and now they are the state of the art model for machine translation and object detection. Just because they aren't useful today doesn't mean that they won't be useful.

The virtue of the Atari games is that they can run at 500 FPS per core, so you can train an RL agent in an hour on a small cluster. When working on fundamentals of the algorithms, being able to iterate quickly makes a huge difference in overall progress. Of course, the idea is that once the algorithms work well we can apply them to real problems, where they'll take months or years to learn something valuable.

'Very few examples of valuable applications to society or the economy.'

Not true. RL has been applied often in production system inside the bigger Internet companies. They are just not published.

Maybe because RL research is not democratized enough to practionners. It is easier to study RL in a controlled environment like a video game (those 'elite researchers' are still not smart enough for real-world applications).

So you remark means that more education about RL is needed, and OpenAI, alongside other institutions like FastAI or Startcrowd, is helping for this effort.

Random search is 10x slower now, and can never be better than that. Also, the ES paper you refer to is still reinforcement learning. This is like comparing neural networks to SIFT+SVM in the 90s. Sure, the SVM performs better, but maybe investing more in complicated models would eventually lead to better results?

Reinforcement learning has been used in a lot of valuable applications to society/the economy outside of games: control system optimization, robotics, ad targeting, content personalization to name a few. Game playing can often be a great test-bed for RL algorithms that can be applied in other areas.

In what circumstances is it only 10x slower? Random search is totally useless when your environment is stochastic. These algorithms aren't learning sequences of actions, in fact most use a 30 'no op' random start to avoid just that.

By 'random', he means evolutionary search. It's not really random, and is just a slower method for policy gradient. Here's the OpenAI blog post: https://blog.openai.com/evolution-strategies/

Reply on news.ycombinator.com