Hacker News Reader

Comment by ldoughty | Hacker News Reader

Really cool! But right as it was nearing 4,000, it seems to have corrupted itself and no longer got any scores above 0. Not sure if that's a code bug or a neural net issue.

avg500 -4.6 last 500 episodes

peak 3959.3 best window

roll/s 20.68 20-step avg

progress 4388 562749 episodes

View on HN

Yes it just collapses eventually — never stabilizes. The training process is flawed, I suspect it has to do with the fact that some weights blow up over time, you can see in “weights” tab.

But at around 4K avg score you should see it solve the env almost every time.

Just a demo :) optimized for speed over stability.

Reward structure: Step: -1 Dot: +100 Win: +1000 so ~4k is max theoretical score on 6x6.

maybe because it doesn't understand "done"? perfect play is impossible, random variance will cause scores to drop even if the model plays well and "wins". feels like it would get stuck in a loop trying to improve what can't be improved.

The optimizer doesn't need to understand anything it's just an iterated mathematical construct. The author simply didn't bother to implement the necessary details to ensure numerical stability.

Alternatively it might be a problem with the scoring model in the end game.

That is what I thought op was saying when he used the word "understood". No need to jump on people using every day language that is still easily understood in context IMO.

feels like it would get stuck in a loop trying to improve what can't be improved.

That is the point, there is nothing on an intention that we cannot improve, the goal here is no more than 1 unique iteration of the same path

I think I noticed it reach “end game.” The snake reaches a point where, if it gets any longer, it is out of squares and hits its own tail. So it finds the route through the squares that it can infinitely loop, never eats the ball, and score starts dropping and goes negative.

Reply on news.ycombinator.com