ConvNetJS Deep Q Learning Demo (2013) | Hacker News Reader

ConvNetJS Deep Q Learning Demo (2013)

cs.stanford.edu

64 pointseigenvalue27 comments

This is related to the Q* stuff that is making the rounds now. I was quite impressed with this when I first saw it years ago. The little 2D robot is able to learn sophisticated looking behaviors with just reward based reinforcement. If you find this interesting, you should also check out Karpathy’s ScriptBots project from back when he was a college student:

https://youtu.be/RjweUYtpNq4?si=IGUwl7cZ47V2DSGB

Interesting that back then he still had that mix of (a) is genius (or at least very smart) and (b) can and is willing to explain things in a way that is entertaining, simple, fun.

He has been a legit superstar since college. No one was surprised that he went on to do great things.

Absolutely. I wasn't saying it that way round. More that the presentation skills were as impeccable back then as they are now. Which may not be surprising but it made me happy to watch. He is one of the few "wormhole" people who understands the hard stuff, but can teach some of it to the rest of us!

P.S. username checks out.

I stumbled across ScriptBots when I was a kid, and became obsessed with implementing my own variants, and subsequently with neural networks. This one [1] is close to Karpathy's original, and this one [2] turns the world 3D and the beings are slightly more advanced. Both are implemented in MoonScript/Lua and run with Love2D. In part due to this this YouTube video, I work as a Machine Learning Engineer today.

[1] https://github.com/nilq/genesis [2] https://github.com/nilq/plane

Yes, I also became somewhat obsessed with ScriptBots when it first came out. I remember re-compiling it from source using the Intel C++ compiler and getting a pretty massive speedup on my machine. It was the first time I really dug into a fairly complex program. Even now I think it's probably too heavy to work well in JS.

When I was in grad school and implemented Q-learning for the first time, was the moment when machine learning "clicked" for me. It went from being a "oh this algorithm is pretty cool, but it doesn't learn anything" to "oh my god, the program figured things out on its own."

I've always had a soft spot for q-learning ever since :)

edit:

In grad school, I implemented q-learning to learn strategies for playing games (like tic-tac-toe, towers of hanoi, etc).

For a very high-level, visual introduction, see:

https://mlu-explain.github.io/reinforcement-learning/

I found that helpful, thank you for sharing it.

This is incredibly interesting. Why wasn't this seen as fruitful over the last decade?

My understanding (not an expert) is a lot of problem domains have very sparse / infrequent rewards - imagine if the only reward you gave a minecraft agent was when it mined a diamond, it would take a lot of gameplay for it to randomly do that and get a reward. So researchers spend time tuning the reward space (oh you mined some dirt, here's a tiny reward. Oh you mined rock, a greater reward, etc) but it's kind of akin to hand crafted feature detection from the pre-neural network days. The Q* mystery is did OpenAI 'solve' reward modelling the same way neural networks solved feature detection.

Reward for a successful prediction against a goal, then the nuance is defining a goal?

Sounds like the process of tuning the reward space is a type of labelling and ranking problem. If I'm not mistaken, those are two things that GPT-4 is pretty good at. You wouldn't even necessarily pre-label every possible action since GPT-4 could do it in real time.

It was. Around the time this came out, something like half of the new ML papers were about reinforcement learning. The problem is that it’s incredibly slow and inefficient compared to any learning where you have access to a gradient and can use that to choose more targeted weight updates.

But there are certain applications where it’s the only good way of doing it (for example in games, where you don’t have access to a gradient over the space of how good a certain move is given the current game state, and it’s relatively quick and and efficient to simulate the evolution of the game).

neuroevolution strategies are another approach to games, for what it's worth (since you said 'only').

For those of us like me who are unfamiliar, can you recommend any useful reading on the topic?

Yes, take a look at Ken Stanley's web presence:

NEAT: https://www.cs.ucf.edu/~kstanley/neat.html

HyperNEAT: http://eplex.cs.ucf.edu/hyperNEATpage/HyperNEAT.html

There are explanations, links to the research papers, and links to implementations.

Here is a working example that runs in your browser using Javascript: https://liquidcarrot.io/example.flappy-bird/

Well Q learning has been incredibly fruitful over the last decade. If you want to know why it wasn’t fruitful before the last decade, the answer is that it was. Data driven techniques went in and out of vogue for this kind of stuff. The difference this time is how much data we have due to the internet and smart phones and now we have computers capable of drinking from the data fire hose

But the recent breakthrough in AI isn't based on Q learning, but on self-supervised learning. Reinforcement learning is only used for part of the fine-tuning of those SSL models.

I don't know much about RL but I was wondering has anyone tried the opposite? Like have a fixed set of actions, and a fixed/ranged movement speed, then a punishment every time it doesn't reach the goal? Or does this not work?

That’s just a negative reward, so basically the same thing. You train most efficiently with a mix of positive and negative reinforcement, just like with children.

Thanks, I can't write a full reply now (need to think) but for some reason my intuition was, let's say you have a constant punishment signal, and a timer, and if it doesn't solve whatever problem by the time the timer goes down, then it has to find the optimal action set, and if it reaches a goal earlier than the timer then it has to weigh that solution stronger? Like at least how I see with organisms it's about finding the optimal use of the limbs in order to solve an ongoing problem/goal state, and if you accumulate specific actions (instead of one network that optimizes one "space"), and different types of goal states, then it has to find the optimal set of actions to reach the different goals, it just seemed more efficient. But this is off the cuff a bit right now.

Edit: i think what i'm thinking of are two "global" numbers. A "closer to the goal" number (higher when closer), and a countdown timer, and then it has to maximize those within the above setting

if you only have negative rewards the bot will usually stop picking any actions at all. bc they all have reward values too low.

sometimes a big negstive reward can scare it away from ever pressing the scary button ever again

I don't think these nets are sophisticated enough to understand complexities of what humans call punishment / reward. Therefore punishment = -reward, but this is not the case for humans. If punishment for example is prison, is reward making people "even more free" for example giving them a flying car or something :-). If money is a reward (or punishment a fine), why are people not optimizing every last cent.

“Reward” in RL is just a real number that tells an agent how well it’s doing. If the reward is negative that could be called punishment. Importantly the agent is choosing actions to maximize (expected estimated) reward, so only the relative reward values matter. So if an agent is choosing among exactly three actions and it knows they’ll give rewards [0, -1, +1], the agent will behave the same as if the rewards were [-100, -101, -99].

I implemented Q Learning in University. Only very basic, sort of like the Mario style Q-Learning. It's quite intuitive, as a concept.

You do need to know the expected outputs - positive rewards and negative.

if openapi is doing q* stuff, then Video by AI Explained sounds most plausible to me.

https://www.youtube.com/watch?v=ARf0WyFau0A

Reply on news.ycombinator.com