Microsoft lets you crash drones and robots in its new real world simulator | Hacker News Reader

Microsoft lets you crash drones and robots in its new real world simulator

theverge.com

228 pointsmnoeld74 comments

Sort of related: Drones as a service (DAAS) - let's you fly real drones via your browser. http://cape.com

Woah, this is super cool! Very interested in how they deploy all these drones, keep them updated and maintained, and (if they are) up 24h/day.

I was just predicting we'll see this some day, and here it is already, amazing.

Very cool idea. That challenge to try to crash one of their drones seems pretty enticing...

Impressive. I wonder how hard it has been to code something to avoid drone crashes.

Was wondering the same, the FAQ says:

  > What if I crash the drone?
  
  You can't! Our engineers have built in extremely robust   safety features,
  such as mid-air collision avoidance and advanced geofencing,
  that makes it impossible for you to crash our drones.
  You’re welcome to try to crash them :)

Does Cape allow independent control of the individual rotors? If so, I wonder how well the safety system would handle a drone being flown to its maximum altitude and then having the rotors on one side run in one direction and the other half of the rotors run the other way. In my mind this would cause the drone to rotate around a vector in the horizontal plane while gravity would be pulling the drone downwards. Higher altitude would give more time to build up the angular velocity of this rotation. If then the angular velocity is high as the drone approaches the ground, the system might not have time to recover stable flight. Thus the drone might crash.

Yep good point! We currently do not allow our users to actuate individual rotors to this degree of control. Instead, we allow lateral translation to move forward & backward, strafe left & right, ascend & descend, yaw the drone, and pitch the gimbal. Basically, standard WASD + mouse game controls.

Most quadcopters don't allow for the motors to be reversed - it's very rare for the motor controllers to support this unless the quadcopter is specifically designed for inverted / "3D" flight with custom motor controller firmware and bidirectional / 3D props.

"What motor inputs would cause the hardest recovery situation" is a pretty fun question, though - I think the best you could do would be to throttle the two motors rotating in a given direction to zero (both CW or both CCW), which would induce instability in multiple axes at once. But there's probably something more creative I'm not thinking about!

The obvious next question: where do I buy such an uncrashable drone for my own personal use? Is there something like that readily available on the market, or is that the company's secret sauce?

So, how exactly does this work? If I'm flying a real drone, in a real location, then how does your system stop me directly flying into a wall/into some other hazard at high speed?

Is it like a really advanced self driving car with sensors that detect solid objects and automatically steers around them?

Because with that quote from the FAQ, part of me wants to see the result of flying the drone full speed at every solid object in the vicinity. Or see how far it can go before the battery dies...

Careful. The "Fly Now" button makes it sound like you're going to put your email in and get to fly. In reality, you're signing up for their mailing list for when they get to Beta....

Behavior like that is why I always give out https://www.mailinator.com/ addresses instead. Or one of the infinite number of alternatives that aren't blocked on site X.

That's true but for me, I signed up perhaps a month ago and was given access to fly just a few days later IIRC. It works really well and is a huge amount of fun.

[co-founder of Cape here]

We're already in beta! But since our system involves physical drones located in the real world, we have limited supply of available flights. We currently load balance the demand from our users by having a waitlist from which we whitelist folks on a weekly basis, and a queueing system when all of the drones in our fleet are simultaneously in the air.

It will be awesome if you mention this just before asking new users to login with google/facebook.

Stopped me from signing up. It's a bit deceptive to say "Fly Now" only to, well, not be able to. I also refuse to log in with Facebook or Google.

Looks cool though...

Fake "actions" on web sites, used to gauge interest, are disrespectful, and quite simply a jerk move. Just like those "sign up for our spam" pop-overs. There are enough things broken on the web without intentionally adding to them. Please don't be a jerk.

I agree, this is a dark pattern.

I wish I have read this comment earlier. Clicked 'Fly Now'. Logged in using my google account and got presented with two huge textboxes. I felt cheated there :D then closed after few seconds.

I'm a member of this team. Please feel free to ask any questions!

Is the benefit of making it a separate tool related to making better/more APIs or avoiding graphical shortcuts made in video games? Or other benefits?

I realize that Microsoft wouldn't have access to GTA5 source code, but Microsoft Studios could have other games, that have already made substantial work in terms of graphical representation and performance, thus saving time for more API work, if the platform could be reused.

One of our major focus is to use AI in real world. A lot has been done in applying AI to video games but we are still taking baby steps in applying it in real world. The question that we want to answer is this: can we train and test algorithms in simulated world and then use them in real world? This is known as transfer learning problem and its a huge area of research where people are investigating, for example, how can we use 80% simulated data and 20% real world data? The games typically found on commercial market are created for entertainment purposes. For example, game creator might decide to cheat on physics to optimize performance and may configure things so its easy but not too easy. Our focus was rather to pursue good physics, sensor models, vehicle dynamics, hardware-in-loop and so on. Also, we wanted to make this project as open source. Most commercial products typically involve complex chain of 3rd party licenses that almost make it impossible for them to be open source.

This is a plugin for the Unreal Engine.

Are there plans for implementing weather simulation? Things that might force course adjustments or reduce visibility would be a useful addon to the simulation. I know one team when I was in school would have loved something like that when they were making a drone meant to scout forest fires and navigate through smoke.

Yes, that's on our plan. Stay tuned. Please also feel free to join our FB group if you had like notifications: https://www.facebook.com/groups/1225832467530667

This would be very important for lighter than air drones. Which... might have much more commerical application anyways due to their long loiter capability.

Can i build the project on a Mac? I only see instructions for Linux and Windows.

Super interested in all things flight (I'm a private pilot and software developer), so I'd just love to get my hands on this.

We are not officially supporting Mac yet, but contributions are welcome :). Our code is cross platform and given enough efforts, I think it can be made available on pretty much any platform. One thing to think about is GPU requirements. Even moderate environments in Unreal can easily require GPUs with at least 4GB RAM.

Many students of the Udacity Nanodegree have found out the hard way that using a stock MBP for CNN training isn't exactly great. Even MBPs with the NVidia 750m struggle (my home workstation is Ubuntu 14.04 LTS with a 750ti SC - and it is only barely adequate).

Honestly, I can't see how you'd be able to run a high-quality 3D sim while training a CNN or other ML model at the same time without something beefier (I haven't tried it yet with my 750ti - but with only 2 gig I have a feeling it wouldn't like it much; it's ok running with the trained model, though).

What students usually (I think) do when all they have is an MBP (especially one with only intel graphics) is train on an AWS GPU instance, then run the trained model locally. This class has been instructive for me in showing that for anything beyond the toy stuff we're playing around with, you really need a beefy ($$$) system. I wish I had the kind of money needed in order to do so (an 8-way 1080 or TitanX box would be sweet - fat luck with me getting it past my wife, though!).

No questions, just a big thanks for doing this, and even more thanks for releasing the source code. Also, thanks for doing it in UE4. This could be very useful to me where I work.

We would love to connect and see how community is using this. Join our FB group and we can take it from there: https://www.facebook.com/groups/1225832467530667

are those 4 views on a single view in the video real time?

That's correct :). The NVidia 1080 yields FPS of ~23 with all 3 views in real time!

When the readme says that the official builds for Linux will arrive in a few weeks, does that mean that it is not yet possible to compile from source also on Linux or is that already possible?

We are internally able to compile the core (AirLib) however there are few rough edges and also we want to make sure Unreal also works as expected on Linux with our plugin.

Is this Microsoft's answer to OpenAI Universe?

No, we are not OpenAI competitors! In fact, we had very much like to work with OpenAI and think about how do we integrate together to advance the research.

Was about to ask, do think this could be use as one of the open AI environments?

Very cool, have you measured the model using real flight data and seeing the discrepancy between the model generated photo/sensor data and the real photo/sensor data? And is their a plan for laser tag mode where multiple vehicles compete with each other for air dominance :)

We are conducting experiments for transfer learning. Stay tuned! For laser tag mode, we will leave it to the community :).

Is this a whole new thing or the evolution of Robotics Studio and the CCR? It looks strangely familiar!

This is not Robotics Studio. It's a simulator for training and testing deep learning/reinforcement learning algorithms for autonomous vehicles. However we are sure community will find other uses :).

What depth camera are you guys using? and at what framerate and resolution?

Have you plans of emulating real hardware and running something like Betaflight[0] on top of the emulated hardware?

Betaflight is the most popular firmware for racing drones, so it could be cool to run that, and test different configuration changes.

[0] https://github.com/betaflight/betaflight

Yes! We have been fortunate to find few in-house drone racers and learning the grit behind their skills. Getting SITL/HITL with Betaflight and/or Cleanflight is definitely one of our important goal in coming months.

Can this be used for more than just multirotor drones? IE - what about other drone aircraft, as well as ground and water-based craft?

I'm currently working thru the Udacity Self-Driving Car Engineer nanodegree; as a part of the first term one of the projects used a Unity3D-based simulator to train a CNN to drive a vehicle.

It would be nice to have a more generalized simulator for this kind of work, to be able to simulate potentially more accurate environments (off road, especially). I could also see interesting work being allowed for underwater ROV environments as well...

As it is, I have considered (for an off-road and more general use) re-tasking TORCS or Trigger Rally (though after just doing some research, it seems like there are ton of these simulators - so maybe one more isn't necessary).

Still - I'm curious whether other environments and vehicles are being considered...

EDIT: It looks like Udacity recently open-sourced their simulator (one thing I have noticed about this course is that sometimes they do things, but don't let the students know about it - that, or maybe I've just been too focused on my work to notice, which is a very real possibility!)

You are absolutely at the right place. We have gone to extra length to design our code so that vehicle specifics are abstracted away behind interfaces. Also things are easily switchable. For example, don't like our physics engine, bring your own! As mentioned in release notes, we are working on other vehicle types. However water based crafts are not on our radar. My guess would be that physics would be more expensive and less accurate in general for water based vehicles.

There ya go https://universe.openai.com/

I have been scanning through the repo for the past hour and I am blown away. Amazing work! As a recent CS graduate who's dream is to work on drones, virtualization and autonomy, is there any place I can shoot you a resume?

1. Do you have plans to adapt this to Autonomous Cars anytime in the near future as a potential temporary work-around for navigation / technology shortcomings?

2. Why not port this in partnership with Forza to make race tracks out of real world environments? I had an idea along this line, though it might be way out of scope heh:

https://hackernoon.com/dashcam-google-maps-dev-kit-custom-ne...

Awesome work! Wondering if it is possible to get data (telemetry, IMU, etc) from a drone during simulation? And is it possible to have multiple drones in the simulation?

Yes, we are supporting to get telemetry as well as ground truth. We also support multiple vehicles but you will need to then build your controller for them or set up multiple HITL/SITL flight controllers.

That's awesome! Can I shoot you an email for more information? I'm doing some research where we're trying to improve flight time with IMUs (GPS-denied environments) and AirSim looks like a great environment for us to get more data and train our algorithms!

Will there be ROS support? So I can use it as a drop-in replacement for Gazebo?

From the actual release page :https://www.microsoft.com/en-us/research/project/aerial-info...

Looks like it already does support it.

So i build / compiled the software, if i understand i need a PixHawk to make it work.

Is there a setup the run the entire project without hardware?

You can and I'd set that up at least once successfully but it isn't easy. I've created a doc for this here: https://github.com/Microsoft/AirSim/blob/master/docs/sitl.md

As someone who works with autonomy for UAVs, previous simulation tools such as gazebo etc. are great for developing and testing control and path planning algos. However, developing, testing and tuning perception techniques such as visual inertial odometry, vision based slam remains and challenge since it typically requires getting your hands on real world data. Photorealistic rendering envs should make it a lot easier for developing good perception algorithms faster. The tools looks very promising, thanks for sharing it and keep up the good work.

First I want to say, "you folks are awesome".

Suppose someone found a great algorithm for drone control based on your simulator and decided to mount a gun on the drone and kill a whole bunch of people.

With all the hotness in AI, rich simulations and algorithms, what steps are you taking to ensure power doesn't fall into the wrong hands?

GitHub repo: https://github.com/Microsoft/AirSim

Neat but this makes me sad because I remember Flight Simulator and how Microsoft killed it.

Quite the contrary - it's alive and well! Just under a different company and/or name.

Flight Simulator X Steam Edition is available on the Stream store.[1] It's the FSX you remember, but upgraded a bit. It's published by Dovetail Games now, since Microsoft sold the "consumer grade" version of their Flight Simulator to them a few years back.

Prepar3d[2] is owned by Lockheed Martin (the defense contractor/aircraft manufacturer), and is what was called ESP when Microsoft owned it. ESP was Microsoft's "professional" Flight Simulator, with enhanced mechanics and military support (missiles, bombs, etc). It was meant to be used in real world simulators used by the military and FAA certified flight schools. However, ESP was sold to Lockheed Martin, and you can pickup a copy of the rebranded Prepar3d which takes FSX to a new level of realism.

I personally fly Prepar3d these days.

[1] http://store.steampowered.com/app/314160/

[2] http://www.prepar3d.com/

I'm a pretty big fan of DCS world, though it's more military oriented.

DCS World is fantastic!

But you're right, it's more military oriented, and I don't know just how realistic the realism is in that game (as far as flight mechanics go, etc). DCS World isn't certified for FAA simulators, for example.

X-Plane is another great flight simulator as well. The siming world is alive and well!

I believe the Prepar3D license forbids personal / private use as they only sell commercial licenses. Something like that.

Also there is X-Plane!

> I believe the Prepar3D license forbids personal / private use as they only sell commercial licenses.

While technically none of the licenses support "entertainment purposes only" (a product of Lockheed buying ESP and not the consumer flight simulator) - one can purchase the Professional license (or Academic license if you're a student) for "training, instruction, simulation, or learning"[1]... which just about covers all the bases.

Nobody said you can't be having fun while learning and simulating!

[1] http://www.prepar3d.com/product-overview/prepar3d-license-co...

This reminds me of Gazebo, http://gazebosim.org/, which I used extensively in college for scenario simulation when working on robotics projects, especially great when used alongside ROS :).

I really thought this was an Unreal/Unity plugin at first. Now that I have read the article, I wonder does anyone know which 3D engine they are using (Rendering, Physics...)?

According to the README it is based on unreal...

"AirSim is a simulator for drones (and soon other vehicles) built on Unreal Engine."

It is Unreal plugin! https://github.com/Microsoft/AirSim/blob/master/docs/build.m...

The Fly Now button signs you up to a waiting list.

looks like unreal engine and the "modular neighbor pack" asset by dokyo. https://www.youtube.com/watch?v=b5CKj0gBo2c

So, jerky gta 5 with an api

Reply on news.ycombinator.com