Hacker News Reader

Comment by Buttons840 | Hacker News Reader

I'd like to ship a video game that does machine learning and trains on its experiences with the player. Yes, I know there are many potential problems with this.

What is the best way to ship training code in a game? Do I embed Python and PyTorch or something? Do I code my own NN training algorithm? Do I use a library such as Tinygrad?

View on HN

If your game has enemies, you could give each unit its own brain. Behind the scenes you could have a "gene pool". When a unit dies it's removed from the pool. When it performs well (eg. damages player or wins the game) the unit can reproduce. Then you add the usual stuff (mutate parameters, cross-breeding fittest units).

For training you could just make two AI teams play against each other (off the cuff, 10-100k games should do the trick). Once you release the game, you could sample the best AI from each player's machine. Free distributed neuroevolution cluster ;)

I've wanted to do this for a while but I haven't made any proper games yet. Someday!

There's actually alot of research in this topic. You should look into "Genetic Programming" on scholar.google.com and you'll find some good presentations.

For example https://scholar.google.com/scholar?hl=en&as_sdt=0%2C21&q=gen... and that references a paper where they do this with MUGEN (Ai v Ai fighting game) http://irep.ntu.ac.uk/id/eprint/30021/1/PubSub7423_8186_Mart...

Genetic programming sounds like a complicated term, but it's basically an easy way to take successful characteristics and breed them into something else.

Typically, these libraries are unusable for this purpose because you don't want to ship a Python interpreter with your video game.

I usually prefer to to rewrite my training step as a pure function, so the model weights are just inputs and the gradient updates are outputs.

You need to serialize your computation graph in some way, so it can be run in C++ or some other low-level language. TensorFlow is known for doing this well since it was original design goal of the project. Some of the other frameworks that originally targeted researchers make this harder. Most mature frameworks have some way of doing this now, though, and projects like https://onnx.ai/ may solve this in general.

It gets more complicated if your model has dynamic control flow, but you get the idea.

It's likely worth looking into federated learning. You could have a core model, which then trains locally on player inputs. The federated learning research space is focused on this sort of small model update on phones, so should have plenty of people who have thought hard about the problem of keeping binary sizes small and training compute manageable.

https://arxiv.org/abs/1909.11875

You probably want a lightweight reinforcement learning library in C/C++ e.g. something from here https://github.com/search?l=C%2B%2B&q=reinforcement+learning...

You could also compile a neural net into a less python-tied format e.g. ONNX or torchscript. In general a siloed pytorch env would be massive, I'm assuming at least a gig or two.

I suppose you could collect training data during gameplay and process it after the fact. Then you can use the heavy python framework, but it doesn't get in the way. Have the user run the training program on the accumulated data whenever they want to increase the level of customization.

In theory, once could use Arraymancer to build the tools to do this, and not have to ship a Python interpreter. That said, I've not used it in anger.

https://github.com/mratsim/Arraymancer

I think video games already do this; video game AI existed before deep learning, it just used things like decision trees and MCMC.

Doesn't forza the racing game do this?

Edit: spelling

Do you know how to program? I have no idea what you're asking.

Yes, I know how to program and know a good amount about machine learning in Python. The problem is these machine learning frameworks are rather heavy to be shipping around, although I guess you could do it. I was wondering if there's any lighter weight machine learning libraries, which might not have all the bells and whistles, but would be better to ship and train models on consumers computers. Most of the time people ship pretrained models, but don't actually update that model on the end users computer.

Client side training with the usual python stack sounds like a configuration nightmare. Would it try to use the users GPU if it exists? Would it fallback to CPU? Does your user need a specific GPU card to play?

I think the real issue here is with the training speed. Presumably a GPU is required to run the game, but is it restricted to just NVIDIA GPUs/a subset of those?

I don’t know of any such libraries offhand and my guess would be that’s because the size of the library generally matters less if there’s a requirement to have a powerful GPU.

Reply on news.ycombinator.com