Python plays Grand Theft Auto V

Can I ask "where have you been all this time" - Do you have some other aggregator or blogroll that you follow? Do you try to avoid time wasted surfing different articles and comments? Did you know about HN before and avoid it due to its reputation, etc?

As the saying goes, Make it work, make it right, make it fast.

In the first term of Udacity's course, we had to implement something similar as part of a project - but it was a later project that was most interesting.

We were given this tool:

https://github.com/udacity/self-driving-car-sim

...though at the time it was closed source and only had a couple of tracks. Anyhow, the idea and goal was to extract frames from the simulator (it provided for this) and using Python, OpenCV, TensorFlow and Keras - develop a neural network to guide the vehicle around the track. For the project, speed was fixed in code, and we only had to worry about the steering (though the output of the simulator also provided for accelerator and braking values as well - so we could use them if we wanted to).

The simulator had three "cameras" - left, right, and center. You could drive it around the course (using a keyboard, mouse, or joystick/pad), and it would dump these frames to a directory, as well as a labeled dataset of the frame filenames and the various datapoints.

So - you had a labeled dataset for training a neural network - input a frame, train it to output the proper values for that frame (steering, brake, throttle). The left and right frames (with other adjustments) could be used to simulate drifting near the left or right side and counter-steer away (to prevent going off the road). Other things we did was to augment the data by adding fake shadows, shifting or skewing the images to simulate turns or whatnot, altering the brightness of the images to simulate glare, etc. So you could get a very large dataset to play with; more than enough to split up into training, test, and validation sets.

For myself (and more than a few of the other students), rather than try to implement a custom network to handle the task, we instead implemented a version of NVidia's End-to-End CNN:

https://images.nvidia.com/content/tegra/automotive/images/20...

Some of the details about the architecture you have to make some educated guesses about, plus we had to scale our images down to reduce the number of input parameters to the network (but if you have the memory, go for it!). I ended up using my GTX 750 ti SC for my TensorFlow CUDA (running everything under Ubuntu 14.04). This has about the equivalent power as NVidias embedded Jetson PX car computer platform (better for running the model on rather than training, but I don't have such a real rig yet).

It worked really well. I'd imagine that if you did the same in GTA V, which has more realism, the resulting model could be used as a foundation (transfer learning) to further train a vehicle for real-world driving. I suspect that Udacity's simulator has three cameras because the self-driving vehicle they developed has the same setup (I'm not sure how it is going to work, but the final term project is supposed to be related to the real vehicle in some manner - they claim our code will be ran on the vehicle, but we'll see how that goes).

So...you might want to give that a shot! As some advice here, I found that for Udacity's simulator, using a steering wheel controller system (with pedals) was more natural and generated better data than using a keyboard/mouse/joystick. Also, while it sounds like implementing this kind of system is simple, in reality you might find yourself being frustrated by certain aspects (make sure your dataset is "balanced" - I found issues in one of my runs where my model was biased to left-hand turns, and couldn't cope with a right-hand turn, for instance) - but these can lead to interesting and humorous results:

I had one model which got confused, and would turn off the road in the simulator and drive on a dirt area; this dirt area was defined, but "edges" consisted of berms and warning signs; other than that it looked the same as the ground. The dirt area wound around then eventually led back out to the track.

My model, driving the car, would invariably go off-roading on this section - but despite never having trained in this area, it drove it relatively perfectly! It would follow the dirt around, and get back on the track, at which point when it got to the dirt entrance again, it would do it over. It did this several times, and was fun to watch. Somehow, the CNN model had enough knowledge from on-road driving to handle a simple case of off-road driving as well...

So you might want to give this option a shot in the future. I personally have waiting in the wings a machine that I planned to build for running GTA V, based around a GTX 970 - it was going to be a gaming machine, but now I am seriously considering making it my next workstation for machine learning (and perhaps upgrading to a 1080).

Have they reversed that trend in V?

So I'll fiddle for 10-15 minutes, occasionally pausing to write small notes and important details down (like the CLI commands I used to achieve some particular result), but only after I achieve some significant amount of progress will I go back and write about what I just did in more verbose language.

The way I have gone about things has changed over time, but I have found the worst way to do documentation is to write 100% of the code, the full series in this case, then go back and document/write a tutorial on it.

I used to just do the videos, but had lots of requests to do write-ups too, so I started that and really hated it at first, because I was just timing it all wrong.

So now, I usually will do maybe 1-5 videos, and then make sure to do the write-ups on them before continuing...otherwise the write-ups suffer significantly. I also try to do the write-ups in the same day, and before I personally progress on to the next topics.

It's just hard, because the last thing I want to do after I've made something that I think is cool is document it.

Other times I will actually work locally, and just either save my scripts in a step-by-step manner, or just work in an ipython notebook to save the steps I took in development, and then film the video, and then go back to the notebook to do the documentation.

Not sure that really helps much, it's an "it depends," but the main thing is to not get too far ahead of any accompanying documentation.

http://www.gamasutra.com/view/news/167781/Indepth_Extravagan...

you could also directly intercept/inject graphic calls like ENB and SweetFX, ENB already supports GTA5

For the Atari/car racing games you get direct pixel output of the game so using a neural network for image processing is required. But other games have a much simpler environment where you can focus on reinforcement learning.

There's even classic doom. https://gym.openai.com/envs#doom

Trigger Rally: This is available as a nodejs/javascript version (arguably the most advanced):

https://github.com/CodeArtemis/TriggerRally

There is also a native Linux version (check your repo - also here: http://trigger-rally.sourceforge.net/).

There are other racing game simulators available for Linux - TORCS (The Open Racing Car Simulator) is another:

http://torcs.sourceforge.net/

Udacity - as part of the Self-Driving Car Engineer Nanodegree, also open-sourced their driving simulator (I posted this link elsewhere):

https://github.com/udacity/self-driving-car-sim

I used it in the first term to implement a version of NVidia's End-to-End CNN (https://images.nvidia.com/content/tegra/automotive/images/20...). Something I ran into with the simulator, which may apply to other simulators (though I've played Trigger Rally before with keyboard only and no problems) is that joysticks were really touchy (major fast acceleration on movement of sticks, and no way to really adjust it easily under Linux - which was kinda a disappointment to see - I could do some adjustment, but nothing like what is available on Windows); I found that using a steering wheel controller worked best.

Poetry.

If you have any ideas: https://github.com/Sentdex/pygta5/pull/3

That said, there are various things like scripthookpy that let you communicate with GTA V to do things other than AI with Python.

I am personally more curious about the AI aspect, so I figure any game that I can play visually, an AI can too with these methods.

At a minimum, I want to be able to send positional data to control the orientation and position of the head, and at least one of the hands (ragdoll physics and IK can take care of the rest). That way, I could control where I look with my head, and where I point or grab with my arm - independently.

Ideally - you'd have both arms/hands free - plus you'd need inputs to control other aspects.

For the head, if you crouch down, it would know this, and adjust the body model as well (or perhaps those parts could be controlled as well). The goal?

Better immersion for VR - but platform agnostic; the thing is, I don't want future games to be beholden to a particular or set of HMDs (ie, this game only works with the RIFT, but not the Vive, etc). I also own a nice older kit of magnetic trackers (an old Ascension Flock) that I could use, along with some older HMDs (as a contributor to the KS, I also have the original DK1, plus the later KS backer CV1 - but both of these aren't well supported under Linux, which ticks me off). Basically, let me (or anyone else) control the character in full, and just provide a way to output two stereo frames (don't even distort them - let third-party software deal with that - because you might have an older HMD that doesn't need the distortion). Then get out of the way.

I doubt that we'll see such a development though.