The History of Machine Learning in Trackmania
hallofdreams.org
hallofdreams.org
Just wanted to provide some perspective here on how many things those projects need to take care of in order to get some training setup going.
I'm the developer behind TMInterface [1] mentioned in this post, which is a TAS tool for the older TrackMania game (Nations Forever). For Linesight (last project in this post), I recently ended up working with its developers to provide them the APIs they need to access from the game. There's a lot of things RL projects usually want to do: speed up the game (one of the most important), deterministically control the vehicle, get simulation information, navigate menus, skip cut scenes, make save states, capture screenshots etc. Having each of those things implemented natively greatly impacts the stability and performance of training/inference in a RL agent, e.g. for the latest version the project uses a direct capture of the surface that's rendered to the game window, instead of using an external Python library (DxCam). This is faster, doesn't require any additional setup and also allows for training even if the game window is completely occluded by other windows.
There are also many other smaller annoying things: many games throttle FPS if the window is unfocused which is also the case here, and the tool patches out this behaviour for the project, and there's a lot more things like this. The newest release of Linesight V3 [2] can reliably approach world records and it's being trained & experimented with by quite a few people. The developers made it easy to setup and documented a lot of the process [3].
[1] https://donadigo.com/tminterface/
I had a few brushes in RL (with collaborators who knew more RL than I did). A key issue we encountered in different problem settings was the number of samples required to train. We created a headless version of the underlying environment but could not make it go a lot faster than real-time. We also did some work to parallelize but it wasn't enough (and it was expensive). Is the TM related RL training happening in real-time or is it possible to speed it up? That seemed like the key problem to make RL widely used, but curious about your thoughts.
We're lucky in case of TrackMania because it internally has systems to both set the relative game speed and also completely disable all rendering and just run physics. Linesight achieves about ~10x speedup where the most time spent now is in rendering game frames and running the inference on the network. They also parallelize training by running more game instances and implementing a training queue. For the "raw" speedup ratios, TM usually achieves about ~60x (one minute is simulated in one second) and I use this speedup to implement bruteforce functionality in the tool (coupled with a custom save states implementation).
> Thus far, every attempt at training a Trackmania-playing program has trained the program on one map at a time. As a result, no matter how well the network did on one track, it would have to be retrained - probably significantly retrained
This is a crucial aspect when talking about RL. Most of the Trackmania AI attempts focuses on a track at a time, which is not really a problem since they want to, given an individual track, outperform the best human racers.
However, it is this nuance that a lot of more business oriented users don't get when being sold on some fancy new RL project. In the real world (think self-driving cars), we typically want agents to be way more able to generalize.
Most of the RL techniques we have do rather well in these kinds of constrained environments (in a sense they eventually start overfitting on the given environment), but making them behave well in more varied environments is way harder. A lot of beginner RL tutorials also fail to make this very explicit, and will e.g. show how to train an agent to find the exit in a maze without ever trying it on a newly generated maze :).
Tangentially related, is anybody besides the autonomous car folks developing games or virtual environments designed from the ground up for exporting machine learning APIs? By this I mean exporting game state and accepting game controls through the network without going through adapter contortions.
I think that point is an important one. ML algorithms work better when they are given better context. Especially in programming, it is clear the models are trained on code, rather than repositories. They know about files and repositories, but i always get the impression that they are totally clueless about whole programs.
What could be done better in code, is provide in training more data about where each function is located in the project, some other files where similar functions are defined or called and so on. In general before each code is fed into the training, to do a little bit of data mining in the project like the tree-hugger project [1] enables. Tree-hugger however is a little bit older code, and tree-sitter has advanced a lot the last 4 years.
In my opinion 5x to 10x in code, is within reach, with no need to increase GPU compute or electricity.
I do love Trackmania. I'd like to play 2020 but alas I do not have a compatible computer. I mostly play the Wii version.
"Of course, we come into this with modest ambitions: make humans obsolete in the process of racing Trackmania. Perhaps we’ll be featured in the cover of a scientific journal to show how AI now dominates all racing sports. The kind of thing we can reasonably expect when entering a project with no directly relevant experience.
Obviously, logically, we know that this kind of oversized ambition isn’t realistic. No one at Google is going to see this and say “They really do need eight billion dollars in cloud credits for the noble task of racing emulated cars.” But having ludicrous ambitions is a key aspect for avoiding the thing that kills most projects: ennui. For us, if a project is merely aiming to be “kind of good” then it’s not going to go anywhere, because being “kind of good” is not exciting. So we trick our brains: we envision the realistic best case scenario (a pretty good racing program) and multiply it by a thousand. Is it grounded? No. But when you’re spending eight hours trying to figure out how to configure virtual desktops over ssh, you need something to keep you going. When every block you put down is a step towards a towering edifice, setbacks feel smaller, and wins feel like they’re worth something more.
So, that’s always been how we approach the early days of these projects, with unrealistic goals that we “know” are impossible. Once its potential starts to materialize, once you have the shape of the work laid down, then you can start to discard the grandiose for the grounded, and then you can start to take those castles in the clouds and boil them down into the bricks of the house. Projects like this are a long haul. We’ve already spent four full months trying to get TM2020 and OpenPlanet both installed. If you don’t have a good goal, you won’t have a reason to make it off the starting line."
It literally doesn't matter if it's 1px or 100 meters to the wall, just learn to not hit it.
Instead, measure from the _camera_. That's all that matters. That's what humans do when we play.
Bonus: with this added perspective you'll be able to drive maps with hills and jumps. Not just flat maps.
At that point you need to run the game real-time instead of faster, so you need a lot of compute to generate your data. You will also need the bandwidth to throw the data around, and the GPUs to take all that input size.
It's definitely possible, but not the place to start, and will require a lot more compute infrastructure.
It would require no extra infra or compute.
E.g. can you improve or replicate missile intercept algorithms