Hacker News Reader

Comment by shitals | Hacker News Reader

I'm a member of this team. Please feel free to ask any questions!

Can this be used for more than just multirotor drones? IE - what about other drone aircraft, as well as ground and water-based craft?

I'm currently working thru the Udacity Self-Driving Car Engineer nanodegree; as a part of the first term one of the projects used a Unity3D-based simulator to train a CNN to drive a vehicle.

It would be nice to have a more generalized simulator for this kind of work, to be able to simulate potentially more accurate environments (off road, especially). I could also see interesting work being allowed for underwater ROV environments as well...

As it is, I have considered (for an off-road and more general use) re-tasking TORCS or Trigger Rally (though after just doing some research, it seems like there are ton of these simulators - so maybe one more isn't necessary).

Still - I'm curious whether other environments and vehicles are being considered...

EDIT: It looks like Udacity recently open-sourced their simulator (one thing I have noticed about this course is that sometimes they do things, but don't let the students know about it - that, or maybe I've just been too focused on my work to notice, which is a very real possibility!)

You are absolutely at the right place. We have gone to extra length to design our code so that vehicle specifics are abstracted away behind interfaces. Also things are easily switchable. For example, don't like our physics engine, bring your own! As mentioned in release notes, we are working on other vehicle types. However water based crafts are not on our radar. My guess would be that physics would be more expensive and less accurate in general for water based vehicles.

There ya go https://universe.openai.com/

Is the benefit of making it a separate tool related to making better/more APIs or avoiding graphical shortcuts made in video games? Or other benefits?

I realize that Microsoft wouldn't have access to GTA5 source code, but Microsoft Studios could have other games, that have already made substantial work in terms of graphical representation and performance, thus saving time for more API work, if the platform could be reused.

One of our major focus is to use AI in real world. A lot has been done in applying AI to video games but we are still taking baby steps in applying it in real world. The question that we want to answer is this: can we train and test algorithms in simulated world and then use them in real world? This is known as transfer learning problem and its a huge area of research where people are investigating, for example, how can we use 80% simulated data and 20% real world data? The games typically found on commercial market are created for entertainment purposes. For example, game creator might decide to cheat on physics to optimize performance and may configure things so its easy but not too easy. Our focus was rather to pursue good physics, sensor models, vehicle dynamics, hardware-in-loop and so on. Also, we wanted to make this project as open source. Most commercial products typically involve complex chain of 3rd party licenses that almost make it impossible for them to be open source.

This is a plugin for the Unreal Engine.

No questions, just a big thanks for doing this, and even more thanks for releasing the source code. Also, thanks for doing it in UE4. This could be very useful to me where I work.

We would love to connect and see how community is using this. Join our FB group and we can take it from there: https://www.facebook.com/groups/1225832467530667

As someone who works with autonomy for UAVs, previous simulation tools such as gazebo etc. are great for developing and testing control and path planning algos. However, developing, testing and tuning perception techniques such as visual inertial odometry, vision based slam remains and challenge since it typically requires getting your hands on real world data. Photorealistic rendering envs should make it a lot easier for developing good perception algorithms faster. The tools looks very promising, thanks for sharing it and keep up the good work.

1. Do you have plans to adapt this to Autonomous Cars anytime in the near future as a potential temporary work-around for navigation / technology shortcomings?

2. Why not port this in partnership with Forza to make race tracks out of real world environments? I had an idea along this line, though it might be way out of scope heh:

https://hackernoon.com/dashcam-google-maps-dev-kit-custom-ne...

When the readme says that the official builds for Linux will arrive in a few weeks, does that mean that it is not yet possible to compile from source also on Linux or is that already possible?

We are internally able to compile the core (AirLib) however there are few rough edges and also we want to make sure Unreal also works as expected on Linux with our plugin.

Are there plans for implementing weather simulation? Things that might force course adjustments or reduce visibility would be a useful addon to the simulation. I know one team when I was in school would have loved something like that when they were making a drone meant to scout forest fires and navigate through smoke.

Yes, that's on our plan. Stay tuned. Please also feel free to join our FB group if you had like notifications: https://www.facebook.com/groups/1225832467530667

This would be very important for lighter than air drones. Which... might have much more commerical application anyways due to their long loiter capability.

I have been scanning through the repo for the past hour and I am blown away. Amazing work! As a recent CS graduate who's dream is to work on drones, virtualization and autonomy, is there any place I can shoot you a resume?

Can i build the project on a Mac? I only see instructions for Linux and Windows.

Super interested in all things flight (I'm a private pilot and software developer), so I'd just love to get my hands on this.

We are not officially supporting Mac yet, but contributions are welcome :). Our code is cross platform and given enough efforts, I think it can be made available on pretty much any platform. One thing to think about is GPU requirements. Even moderate environments in Unreal can easily require GPUs with at least 4GB RAM.

Many students of the Udacity Nanodegree have found out the hard way that using a stock MBP for CNN training isn't exactly great. Even MBPs with the NVidia 750m struggle (my home workstation is Ubuntu 14.04 LTS with a 750ti SC - and it is only barely adequate).

Honestly, I can't see how you'd be able to run a high-quality 3D sim while training a CNN or other ML model at the same time without something beefier (I haven't tried it yet with my 750ti - but with only 2 gig I have a feeling it wouldn't like it much; it's ok running with the trained model, though).

What students usually (I think) do when all they have is an MBP (especially one with only intel graphics) is train on an AWS GPU instance, then run the trained model locally. This class has been instructive for me in showing that for anything beyond the toy stuff we're playing around with, you really need a beefy ($$$) system. I wish I had the kind of money needed in order to do so (an 8-way 1080 or TitanX box would be sweet - fat luck with me getting it past my wife, though!).

Awesome work! Wondering if it is possible to get data (telemetry, IMU, etc) from a drone during simulation? And is it possible to have multiple drones in the simulation?

Yes, we are supporting to get telemetry as well as ground truth. We also support multiple vehicles but you will need to then build your controller for them or set up multiple HITL/SITL flight controllers.

That's awesome! Can I shoot you an email for more information? I'm doing some research where we're trying to improve flight time with IMUs (GPS-denied environments) and AirSim looks like a great environment for us to get more data and train our algorithms!

Have you plans of emulating real hardware and running something like Betaflight[0] on top of the emulated hardware?

Betaflight is the most popular firmware for racing drones, so it could be cool to run that, and test different configuration changes.

[0] https://github.com/betaflight/betaflight

Yes! We have been fortunate to find few in-house drone racers and learning the grit behind their skills. Getting SITL/HITL with Betaflight and/or Cleanflight is definitely one of our important goal in coming months.

First I want to say, "you folks are awesome".

Suppose someone found a great algorithm for drone control based on your simulator and decided to mount a gun on the drone and kill a whole bunch of people.

With all the hotness in AI, rich simulations and algorithms, what steps are you taking to ensure power doesn't fall into the wrong hands?

Very cool, have you measured the model using real flight data and seeing the discrepancy between the model generated photo/sensor data and the real photo/sensor data? And is their a plan for laser tag mode where multiple vehicles compete with each other for air dominance :)

We are conducting experiments for transfer learning. Stay tuned! For laser tag mode, we will leave it to the community :).

Will there be ROS support? So I can use it as a drop-in replacement for Gazebo?

From the actual release page :https://www.microsoft.com/en-us/research/project/aerial-info...

Looks like it already does support it.

So i build / compiled the software, if i understand i need a PixHawk to make it work.

Is there a setup the run the entire project without hardware?

You can and I'd set that up at least once successfully but it isn't easy. I've created a doc for this here: https://github.com/Microsoft/AirSim/blob/master/docs/sitl.md

Is this a whole new thing or the evolution of Robotics Studio and the CCR? It looks strangely familiar!

This is not Robotics Studio. It's a simulator for training and testing deep learning/reinforcement learning algorithms for autonomous vehicles. However we are sure community will find other uses :).

What depth camera are you guys using? and at what framerate and resolution?

Is this Microsoft's answer to OpenAI Universe?

No, we are not OpenAI competitors! In fact, we had very much like to work with OpenAI and think about how do we integrate together to advance the research.

Was about to ask, do think this could be use as one of the open AI environments?

are those 4 views on a single view in the video real time?

That's correct :). The NVidia 1080 yields FPS of ~23 with all 3 views in real time!

Reply on news.ycombinator.com