Pac-Man recreated with a GAN trained on 50k game episodes
blogs.nvidia.com
blogs.nvidia.com
A) It's no longer practical to incorporate into real products given the hardware costs for training and inference. In NLP, large transformer based models can easily occupy a 5 thousand dollar GPU at a paltry 50-60 words per second processing pace.
B) The Research often demonstrates things that existing non-ML toolchains were already reasonable at doing with a better result, and a faster iteration time.
This demo looks like it's replicated the core mechanics of pac-man with worse graphics, 3-4 orders of magnitude higher resource consumption, and undoubtably many odd bugs hidden away from the pretty demo. It even needed the game to exist in the first place! While this is fascinating from the technical and academic perspective, it's hard to be convinced that any gains we see in the field won't be offset by increased hardware/latency costs from a product standpoint.
B) This specific example actually shows ML toolchains are orders of magnitude better, than existing ones. Imagine you wanted to make a football game from scratch, and this technique would let you do that by just feeding a lot of football game recordings to a program, as opposed to hiring software devs and graphics designers, and iterating for a year.
This isn't too bad of a situation if this GPU provides 100% of your value proposition. but, if you needed your product to perform 5 such tasks - or competition/customer demand forces you to move to a bigger model with a 10 words per second inference rate then you may find it hard to balance customer value vs. price vs. margin.
https://twitter.com/br_/status/979442438254166016
> "selling AWS at a loss" is crisp shorthand for a lot of startups' business models!
A bit thin of the ground (especially when it comes to supporting the claim about the usefulness of the memory module) but the way neural nets research and especially RL research is going (from what I've seen, not my area of expertise) that's the kind of result that you can expect will get your paper published these days.
Indeed, a common criticism against papers in machine learning is "what is a real-world application of this approach?". If you can't answer that convincingly then it's unlikely your paper gets published, particularly if it has rather mediocre results otherwise, as is the case with this paper (see my earlier comment about the comparison to the two other systems which is a bit of an anticlimax).
In fact, the paper (well, the preprint version) makes a clear attempt to motivate the work with a practical application, saying that it aims to "learn a simulator [for robotic agents] by simply watching an agent interact with an environment".
This is 100% not true for computer science and I don't know why you think it's true. There's are entire branches of computer science (e.g. complexity theory) that are highly theoretical and are certainly not geared towards immediate practical application.
Even in more applied branches of computer science like ML there are many papers that are geared more towards the fundamental research side of things.
> saying that it aims to "learn a simulator [for robotic agents] by simply watching an agent interact with an environment".
Saying that a paper could lead to a useful application down the line is not the same that saying that the paper is intended to describe the schematics for a specific useful application.
I'm not sure who brought up a schematic? The bit I quoted is taken from the paper and it's a clear attempt to motivate the paper's approach as having a practical application. To clarify, do you disagree with that?
I see a strong parallel. It's a poor performance but important proof of concept.
Also, it's general in computing trend since assembly. Why compile when you can write machine code, waste of resources. Why have abstract compiler, when you can write assembly. Why have higher level language when it cant generate code optimal for target when it doesnt know it. Then web came, it's not faster than 20 years ago because with increased computing power bloat came. Somehow it always turns that seemingly wasteful apporach outweights its cons. Empirically lowered entry cost and increased applicability has very very high value. At least higher that I used to assess.
Also, the paper itself tries (hard) to motivate its approach with a real-world application, in particular, learning a simulator to train robotic agents to interact with the real world.
---
In my field we make robots do a thing, say folding laundry, as an example of dextrous manipulation[1], and the popular conversation is about whether your laundry robot makes commercial sense and not about your novel grasping strategy and nifty motion planner that makes this sorta-possible for the first time. But everyone sees your video. Pros and cons.
[1] laundry folding at Berkeley (not me) https://youtu.be/gy5g33S0Gzo
Sure.
> That is essentially a kind of consciousness.
That is a flying leap.
It just means some people poked a pile of a bazillion if-else statements long enough, until the desired result came out. I believe everyone here knows the xkcd I am referring to.
Still, I think it's an interesting result. But, as someone on Fridman's podcast said the other day: recent advances in AI have all been thanks to advances in computing power, not any new ideas. The future if AI seems to depend on Moore's Law, more than anything.
But eventually we reached the point where they are not only competitive, they may be positioned to fully overtake combustion cars.
I wonder how many discoveries on electric cars ended up being fully dead ends along the path that took us to the point we are now?
The learning part is done by training on the output of an existing simulator. Obviously, if you already have the output of a simulator, you don't need to learn a simulator so the only real use of this approach is to train a simulator by watching a human agent interacting with the environment. So the idea is to eventually apply this thing to training robots to interact in the real world by watching how humans do it (this is not clearly stated in the paper; it's my charitable interpretation of the very poor motivation of learning a simulator from the output of an existing simulator).
The only problem is that creating an accurate simulation of a human's interaction with the real world (let alone an accurate simulation of the real world) is prohibitively expensive. Thus, for the time being, this kind of idea works only in very, very simple environments like PacMan.
So we don't yet have an expensive prototype electic car that nevertheless can do everything an er other kind of car can do. We have... not sure what. The description of an idea with an example of how such a thing could be achieved in the far future, when we can train robots to act like humans by watching us.
I don't see that trend. Bleeding edge research has always been beyond what's practical, by its very nature. The focus is on pushing the boundaries, not to build something efficient/commercial.
It's the followup work that makes it practical. See for example the evolution of WaveNet. It was breakthrough research, but yeah it wasn't practical. Now a couple years later and we're generating speech on mobile devices of similar quality (see the recent Facebook research), and able to teach these machines using orders of magnitude less data (see 15.ai).
Not to mention the plethora of research that is explicitly about making neural networks more efficient and easier to train (e.g. EfficientNet; 1cycle; etc).
Bleeding edge stuff like AlphaStar, BigGAN, GPT-2, etc are just that: bleeding edge. They'll get more efficient and practical every year, just like everything else in machine learning.
Development always starts with some novel technology that has major flaws, but some good nuggets. Then it is further developed to be practical.
Pushing boundries != commercialization potential
Just 10000x? More like 6 orders of magnitude
thinking back on the PC, Web, cloud, and mobile growth periods, each came with a transformative change in the capabilities, cost, and availability of computing power. We're only seeing two of these aspects in AI today - with the cost side heading in the wrong direction.
The english longbow was vastly superior to the first guns that came out, which were slow to load, unreliable to shoot, didn't work in the rain, and were extremely inaccurate.
3d printed stuff sucks. It's not very strong, it's not very pretty and takes a long time to create.
Meanwhile, take a look at this ML photo processing technique:
True, though my first thought was that this would be interesting from a reverse engineering point of view. Then again Pacman is pretty simple concept. Something worth reverse engineering is probably a lot more complex.
I remember that video because it always struck me as not being far off from reality. It’ll make the current set of online game cheats/hacks seem like child’s play if it can be done effectively (read: human-like and undetectable). Maybe then a server-running AI will be needed to detect AI-like gameplay mimicry.
(Conversely, if they're _not_ functionally indistinguishable from a human player, then that means it's possible to write an AI or other algorithm to detect them.)
if the AI is truly indistinguishable from a human player, then it sort of depends on the human player we are comparing to. if the AI plays like the best human players in the world, this would be a big problem for games without a good ELO system.
Because, even if acting like a human, AI can play/fight 24/7. Many online games still have leader boards that can be climbed through steady play rather than short-term bursts of skill.
Here is an excerpt from a now deleted Reddit comment where the setup is described:
> "My pi is connected to my pc as a HID device with all the descriptors matching a legitimate mouse. My actual mouse is not connected to my pc and the movements are relayed through to the pi over wifi (UDP). My pc records my screen and streams info to my pi over wifi which my pi happily adjusts as seen in the video."
Suppose PacMan is in a corridor and there is a dot to the left of him and three to the right: ".P..." That is plausible because PacMan hasn't eaten any dots yet. But if the second dot to the right is missing: ".P._." something is wrong because the first dot to the right is still there and PacMan can't jump over dots. It must be very hard to get the discriminator to understand that the second image is fake.
Interesting scenario, it would be a cool experiment to walk through the latent space and try to find that kind of frame in the GAN latent space. I'm sure you could find it, but the question might be how far off the generated data's manifold is it.
Don't tell me it never happened to you!
That was one of the weirdest experiences I think I've had.
A few times a year I might wake up in the middle of the night and — at first glance — perceive a goblin sitting on a nearby chair. As more of me wakes up, I realise it’s a pile of my clothes.
Now I’m learning German, I have the experience of listening to a conversation in German, thinking I’ve heard a particular word, asking the locals what it means, and finding neither of them noticed that word being used — but I genuinely hear the unspoken word, so I know a pre-conscious part of my mind is filtering all the German I hear through a vocabulary which is too small.
My mother has Alzheimer’s, and one of the bigger surprises was realising quite how many reality-models we have by way of her losing them at different times. When I was caring for her, star jumps terrified her, she forgot what windows look like at night and how to count past 5, and she lost both object permanence and the concept of left: https://kitsunesoftware.wordpress.com/2018/01/31/alzheimers/
One of the advantages of having a model of the world is it allows you to predict the future.
The only disadvantage I can think of is that it is comparable to branch prediction failure in a pipelined CPU — for example it can be very tempting to apply that to fellow minds, and when those minds don’t think like you, you can annoy them no end by telling them (implicitly or explicitly) that you know better than they do what they do and don’t want. Or you might become convinced that wearing lucky pants causes your sports team to win/your lottery numbers to come up. Or etc.
Cut to a few hours later when they took a break and wanted a snack. So they went to their kitchen and tried to look in their real world fridge, and immediately slammed their head into the door as they tried to stick their head through it without thinking...
I have not actually had any incidents, but I have to consciously not crash into things, or drive wildly.
If you manage to not move your eye while paying attention to the finger, you'll find that it doesn't look quite right when you put it over the blind spot.
With your finger you know it should be there, so something finger-like fills the blind spot. If instead of a finger you try with something like a black dot, you'll see it disappear completely.
But maybe if training sample size can be considerably reduced, game making can open up to far less technical people. Instead of developing with an engine, game makers could show the AI "when I press this button, change the scene like this", and the AI could try to figure out the rules from given samples.
That the network apparently has a mental model which is rich enough to simulate actual game play implies that it hasn’t just fit a static caricurature of the game which immediately breaks on an unseen move. That is, it has a non-trivial “conceptualization” of its environment which goes beyond simple pattern matching or nearest neighbor matching.
Meanwhile an actual human will probably need only about 5 games to fully understand how the game works and reacts.
It almost needn't be said that human culture puts a great deal of emphasis on making numbers grow larger, particularly those associated with personal wealth and economic growth. To a computer without this context, why should there be any reason to prefer larger numbers over smaller ones? Heck, if you're going all the way and denying the AI direct access to the game's memory, then it's got to learn to recognize the digits first before it can even begin to decide whether it's good to increase them.
Even more abstractly, the game relies on the human concepts of good and bad. It's good to eat the fruit and pellets to increase your score counter. It's bad to let a ghost catch pac-man, causing him to die and lose a life. Without being taught, human children can identify all of these elements but the computer would seem to have a more difficult task.
& also that humans come with a huge amount of training beforehand. We've experienced eating & ghost stories & moving around in 3d spatial dimensions before playing pacman usually
Early games will establish things like "can ghosts go through each other", "if I hit the wall do I die" (like in Snake; though on Acorn Electron you could eat the wall sometimes!), "can ghosts go through the shortcut", "how long do pills last", do all pills last the same amount of time".
Then later efforts would be needed to know things like "what algorithm do the ghosts follow", "does the same colour ghost always follow the same algo", "do ghost algos change when you eat a fruit", "what order do the fruit go in" ...
Of course you could guess a lot of these, but that's not what "fully understand how the game works" means.
Also, Nvidia's thing doesn't do the fruits and possibly not even eating ghosts or points and probably not the ghost algos.
Impressive still, but not exactly what's lead to be believed by the title.
https://cdn.arstechnica.net/wp-content/uploads/2020/05/Nvidi...
Emphasis mine.
Also, while I disagree that this would be real "fair use" (rather than Internet Pundit fair use, which is virtually indistinguishable from "I want this, so it must be legal") without permission, it wouldn't matter anyhow. If nVidia did this without permission and tried to distribute exactly what we saw, it would also be a trademark violation. Pacman is almost certainly in the top 1% of defended trademarks, if not .1% or .01%... not a great thing to try to use without permission.
Plus there is a massive give away that Namco approved this within the article:
We were blown away when we saw the results, in disbelief that AI could recreate the iconic PAC-MAN experience without a game engine,” said Koichiro Tsutsumi from BANDAI NAMCO Research Inc., the research development company of the game’s publisher BANDAI NAMCO Entertainment Inc., which provided the PAC-MAN data to train GameGAN. “This research presents exciting possibilities to help game developers accelerate the creative process of developing new level layouts, characters and even games.”
Imagine visual programming but instead of connecting systems and dealing with high level schemas, the stake holder just defines the data flow, and the model creates a (hopefully) more extendable system than current visual programming systems.
But the complexities/black box nature of some neural nets (and the CPU designed by AI that was enormously complex) may transcend to this use case. Meaning we can create systems that are more efficient, more easily, but we ourselves can’t interpret the entire system, let alone each piece sufficiently.
The most successful examples I've heard are computer generated classical music, but generating abstract art like classical music is simpler than generating art that relies on coherent language and narrative, like opera, or the blues, or a good novel.
Edit: oh, Togelius also seemed to have researched it.
https://www.google.com/amp/s/www.technologyreview.com/2017/1...
[0] https://www.theverge.com/2017/9/10/16276528/ai-video-games-g...
Eg if you have a blob that can move in x and y coordinates, it doesn't get more complicated just because you move from 32-bit floating point numbers to 64-bit floating point numbers. Even though the number of states squares.
The hope is that the model can combine inputs from different domains and generalize better.
https://towardsdatascience.com/code2pix-deep-learning-compil...