ChosunTruck – Euro Truck Simulator 2 autonomous driving solution
github.com
github.com
*) http://www.newyorker.com/tech/elements/desert-bus-the-very-w...
It's amazing to see a game with the roads of half continent. I've been playing driving games for so many years that I become fed up with all the grinding (licenses, career path, races blocked by other races, etc) so I'm not buying those games any more. I want all the cars, all the roads, no traffic, no police, only driving. I could pay an extra for that. I wish there is something like Euro Truck with only roads, some fast car and curated lists of interesting road to drive on.
And kudos to the authors of ChosunTruck, obviously.
Then you don't want to play ETS2.
> It's amazing to see a game with the roads of half continent
As someone else noted, the road network is only an approximation of the highways. Each city is essentially the same layout with slightly different buildings to make it look like the namesake city (outside the drivable world, basically a 3D skybox).
The AI vehicles are quite stupid. I've had them suddenly brake in front of me, or merge into the trailer from behind. Oh, and you get penalised for this (ticket & damage repair costs).
Apart from the PS of trucks, there's virtually no difference between them. Want a basic Volvo? It'll haul trailers. Want the top of the line Volvo Globetrotter with sleeping cabin? It'll haul trailers. Oh, and you still need to stop at the designated rest areas to sleep, even though you have a bloody sleeping compartment in the Globetrotter!
You have to follow all the laws, all the time. Speeding? That's a ticket. Running a red light? Ticket. Bumping another car? Ticket!
I wish they would include an arcade or "god mode" so you could actually have fun.
/rant
Can anyone comment on how this is actually obtaining the video output from ETS2?
Is there some way you can feed the output of an OpenGL application into this? Or is it just capturing the desktop area where ETS2 is running?
I have, but it's been about 6 months since I played. Maybe they've updated it to add additional options.
I also haven't installed any mods, so perhaps this could be enabled via additional mods.
> I wish they would include an arcade or "god mode" so you could actually have fun.
They do. Look in settings, you can disable vehicle offenses (so no fines for running red light, crashing, speeding) and you can disable traffic too (might need a mod, i dont recall)
When it comes to the city layouts, the DLCs for Scandinavia and France have better and different layouts than the main game.
I'm sure the issue here is that it is "more common than the real world", but you could replace "AI vehicles" with "other drivers" and that would be a valid real-world complaint.
In Windows, it's done by grabbing the handle of the window with the "Prism3D" classname.
This then goes to the hwnd2mat function[1], which does `GetDC`[2] to get the device content and then creates a bitmap with `CreateCompatibleBitmap`[3].
---
On Linux[4], it's done by grabbing the screen with `XOpenDisplay` and the current window with `DefaultRootWindow`. It then uses `XGetImage` and memcpys the result into the vector of pixels.
Note that the `main` function in the Windows and Linux version is slightly different to accommodate the difference in how these images are grabbed.
[1] https://github.com/bethesirius/ChosunTruck/blob/master/windo...
[2] https://msdn.microsoft.com/en-us/library/windows/desktop/dd1...
[3] https://msdn.microsoft.com/en-us/library/windows/desktop/dd1...
[4] https://github.com/bethesirius/ChosunTruck/blob/master/linux...
Also, active on Twitter: https://twitter.com/outerradev
..and then, like with the bots that supposedly roamed No Man's Sky's development, ensuring no random variables came out with anything rude or inappopriate, make bots like that here to check the merged output isn't quirking too much :)
Anyway, cool project! It's really nice to see what used to be very expensive proprietary technologies become available for anyone with an Ubuntu box to play and experiment with.
I rather see it as a tongue-in-cheek name, it doesn't necessarily convey a self-deprecating mood (though it somehow implies the comparison itself).
Still a pretty cool idea to see Euro Truck Sim being used for AI training. I'm impressed.
This is just how the maps are designed. It's only on the rare three lane stretches that you're able to bump over to the middle lane confidently.
These devs should really start/join a DARPA challenge team; try out their ideas on real courses.
Interestingly, that post links to a specific blog post on GTA V integration[1], but going to that link redirects to the initial "Introducing OpenAI" post[2]. I can't find the GTA V blog post[3] on their blog anymore.
Edit:
In addition, the GitHub repository[4] for spinning up a GTA V server instance from the post[3] no longer exists.
[0]: https://news.ycombinator.com/item?id=13375543
[1]: https://openai.com/blog/GTA-V-plus-Universe/
[2]: https://openai.com/blog/introducing-openai/
[3]: http://web.archive.org/web/20170112173923/https://openai.com...
[4]: https://github.com/openai/universe-windows-envs/blob/master/...
edit: An issue was opened[0] but there has been no response. I find it hard to believe a C&D comes with a gag order but IANAL.
I find it frustrating there's no transparency on this, and I hope I'm not being overly dramatic, but I would have expected more openness here.
There could be a very reasonable explanation for keeping mum on this, I guess.
That looks awesome, nice!
It's not trained; it's entirely programmed.
Looking at /windows/src/main.cpp:
1. Grab the game window with `FindWindow("prism3d", NULL)`.
2. Get an image of said game window with `hwnd2mat` (found in windows/src/hwnd2mat.cpp). This part is complicated, as it deals with the Windows API.
2. Apply Median Blur to the image, as shown with this transformation[2].
3. After creating the 4-point correspondence between the input image (`image`) and the output overlaid image, (`outputImg`), create an IPM object that will perform the Inverse Perspective Mapping. This is a function that turns a flat-angle shot into an overhead shot [3].
4. Resize the `outputImg` to 320x240 for the smaller window.
5. Convert `outputImg` to grayscale, storing the result in `gray`[4].
6. Perform the `cv::blur`[5] operation on `gray`, storing the result in `blur`. This reduces noise in the image; there are artifacts gotten from the Inverse Perspective Mapping.
7. Perform the `cv::Sobel`[6] operation on `gray`, storing the result in `sobel`. This gets all of the outlines, including (surprise!) the lines on the road.
8. Perform the `cv::threshold`[7] operation on `sobel`, storing the result in `contours`.
Result at this point: A flat perspective of the road has been turned into an overhead perspective, converted to grayscale, blurred, turned into an outline, and then thresholded to solely have the lines on the road. Note that there's an option to perform Canny[8] as well, which is another alternative.
9. Create a `LineFinder` object `ld` (found in /windows/src/linefinder.cpp).
10. Perform `ld.findLines` on `contours`. This performs the `cv::HoughLinesP` operation[9], returning a vector of lines (`vec4i` is a vector of 4 integers that stores x_1, y_1, x_2, y_2). It'll use these to draw the detected lines for informational purposes.
---
Okay, now we have a black-and-white image of road-line contours, stored in `contours`. Unfortunately, keyboard and mouse input doesn't work in Windows, so we have to go over to linux/src/main2.cc. Note that the line-finding part is the same thing but requires some Linux-specific stuff to get the original image of the game.
Starting from Line 198:
11. Define the center of the image to be `bottom_center` at column 160.
Our x-value for the pixel values is going to be `j`. Our y-value is going to be `i`.
12. Create a for loop that iterates through all values from y = 240 to y = 30. Inside this is effectively two loops - one that acts on the x-values from 160 to 10, and one that acts on the x-values from 160 to 310. This loop looks at the pixel in the (j, i) spot in `contours` and sees if it detects a line there. If it does, and it hasn't found a line yet, it sets the `center_to_right` and `center_to_left` values, respectively. This finds the x-values of the left and right lines. I think you know where we're going from here...
13. Create a `centerline`, which is going to be the average of the two.
14. For informational purposes, gather the average `centerline` values and the average `center_to_left` and `center_to_right` values.
15. Move the mouse according to how far away `centerline` is from the center of the image. I don't know how to move in ETS, so I'm going to take their word for it that their gnarly-looking trig function is on-point.
[1] https://i.imgur.com/cMfkFGP.gif
[2] http://docs.opencv.org/3.1.0/median.jpg
[3] https://marcosnietoblog.files.wordpress.com/2014/02/sample.p...
[4] http://docs.opencv.org/2.4/_images/Load_Save_Image_Result_1....
[5] http://docs.opencv.org/3.1.0/blur.jpg
[6] http://www.mon-club-elec.fr/mes_images/javacvpro/javacvpro_e...
[7] http://docs.opencv.org/2.4/_images/Threshold_Tutorial_Theory...
[8] https://3.bp.blogspot.com/-ybDW9YBG05s/UsvD3HRj7JI/AAAAAAAAB...
I had tried to comment this code in my own branch (https://github.com/ajchili/ChosunTruck/commit/61f184521ac4fb...)
but this actually summarizes a lot better!
And yes, the gnarly trig function is on point.
Thanks a ton for showing a real-world example of how to use OpenCV. The examples given by the official documentation are cute, but they're individual tools that you have to combine intelligently to make something useful.
Right now I'm changing SendInput to use mouse simulation instead of pressing keys. Hopefully this will fix some of the lagging caused by pressing keys too fast / too much.
The only real difference is some of the order (in the project, you first have to fix the camera distortion, then do the image thresholding, then warping the image from "perspective view" to "overhead view") and the lane finding (where you do a "windowing" method to find the lines after doing a histogram search to find the start of the actual lane-lines; after that, you essentially do a polynomial fit of the found pixels, which you then use to render the road shading, then re-warp that from flat back to perspective and overlay it on top of the road image).
Can you tell I just recently completed all these steps...