Teaching a World Model to Play Pokemon
nostalgia.dev
nostalgia.dev
Show HN: Jev Plays Pokémon Red - https://news.ycombinator.com/item?id=49845172
For reference, standard PPO has been able to beat the game end-to-end with a relatively small network https://drubinstein.github.io/pokerl/
You could compose multiple network, one with the goal of learning how to interact with the world through the screenshots. This would probably require a lot of frames of the game from many diverse locations and scenarios--as the network would need to learn how battles works, items (though technically not strictly necessary), and general dialogue and menu interactions.
Then other networks could then try to learn to encode different intermediary goals trained on a bunch of noisy runs that complete that goal. Collecting this data seems tedious and difficult though and is a whole project in itself.
So my intuition is yes, but I don't think simply making my current model bigger would get us there. I see you have experience with world models--what would you try? ;)
And in terms of what I'd try, I think it's a really hard question haha! I'm interested in the idea of hierarchical world models, where you have something doing prediction on a long-horizon, abstract task level (if I beat this gym I can fight the next one) as well as a short horizon model predicting over individual inputs moment-to-moment. I believe LeCun was actually attached to a hierarchical JEPA paper for robotic control - but it's still very early days.
Art is a conversation between two people, that is a timeless communication between human.
An artwork to realize... to surf within to learn from, to listen to, to realize someone's images, someone's ideas, someone's worries, someone's messages... someone...
Art is someone's work of life, their experience, their... love...
There's human time in art... in a book, article, painting, soundscape, film, video-game... there's alive time of other human who reified their life experience, views, ideas... in an artwork and passed it to the outer world... to be inscribed in the infinitely magnificent history...
This...? This is just, sorry, deadly awful, dreadful, dark, and dear pain in heart... This is also disrespectful towards the Artists, Developers, people who worked on the Universe and this exact artwork...
This is utterly sorrowful...