Hacker News Reader

Comment by Permit | Hacker News Reader

One thing that stood out to me is that they train it entirely on simulated data and deploy it without needing any fine-tuning on real-world data.

I took one of Andrew Ng's deep learning courses a few years ago and remember him saying that this approach typically doesn't work. It sounds like that's no longer the case which is really cool because it's relatively cheap/easy to generate simulated data.

Is this a more common approach these days? Are there any other interesting examples to point to?

View on HN

This depends almost entirely on the accuracy of the simulated environment relative to the real world. If we have very accurate simulators then sim2real isn't too bad. If we have poor simulators (or simulators which don't fully capture that <0.1% probability action that happens to have REALLY bad consequences...) then sim2real doesn't work.

Flying a drone through a static forrest in no-wind (no moving branches, etc.) seems like a reasonable sim2real environment, in part because... well... video games have had forrests and trees don't tend to move very fast unless there's wind.

But sim2real for self driving cars is a lot harder given the huge diversity in streets and freak events that happen (people, bicycles, shopping carts, piano-delivery-trucks)

- Twitter: @iamtrask

I'll add that real world sensors (including cameras) are noisy in ways that are difficult to model 100% accurately in simulation. Same thing with aerodynamic effects, especially when the drone gets close to the ground or other objects, or y'know...it's windy outside. Dealing with these types of noise and unpredictably and building in resilience to them is a big part of testing and validating systems like these, and can only happen in the real world.

I'm pretty confident the drone linked in this article doesn't have high reliability. I'd be shocked if they managed to get >70% critical without significant real-world iteration.

It looks from the video someone linked to above that these things don't use cameras, just depth sensors -- maybe they're easier to simulate? It looks like they do introduce noise into their simulated depth-sensor data, but maybe that's more straightforward than creating realistic visual noise.

I think it basically helps that quadcopters are already computer stabilized and highly manoeuvrable so the problem is more identifying spaces to move into and motion planning through them. What’s impressive since the last time this kind of thing was posted is the increase in speed and at least in the provided video the ability to avoid really quite small branches.

The simulation is a bit of a red herring, in this case. What operated in simulation was not the neural net but a different algorithm. The article calls it a "simulated expert" and my guess is that it was most likely a classical planner. This "simulated expert" was given all the information and, crucially, all the _time_ it needed to fly through a simulated environment (or possibly many more than one). The neural net was then trained on examples of the "simulated expert's" behaviour. Then the trained model was loaded onto the drone and was used to pilot the drone, essentially predicting the path the "simulated expert" would have taken in a similar situation.

In short the "trick" is that they generated a plan offline and then learned to approximate it to avoid having to re-generate it for every flight, which is very time-consuming and wouldn't allow fast flight.

It's clever and as the authors are quoted to say in the article, it doesn't rely on a very precise simulation, since the neural net doesn't have to be trained in a simulator and a classical planner doesn't need very granular detail (it's just slow). On the other hand, it's difficult to know how well it works in practice because "flying through a forest" is not really an objective measure, insofar as there is no ISO standard for what "forest" means. Sparse forest? Dense forest? Conifers? Other...? The news item above says that the approach was tested on "previously unseen" environments, but that, too, is a very fuzzy expression. So while this is very impressive news (without having read the original work) keep an open mind about whether this will really work in arbitrary environments in the real world.

The sim2real approach can work pretty well as long as you are very in tune with where your simulator falls short relative to the real world and take steps to circumvent those shortcomings.

Here's some work my lab did on sim2real for a roughly human-scale bipedal robot (Cassie): https://www.youtube.com/watch?v=MPhEmC6b6XU

We were able to train the robot to climb stairs completely by feel/proprioception without any sort of vision. We trained it in simulation, and then transferred it to the real world without issue.

Yup! There are a ton of examples of this in robot learning. For instance: https://arxiv.org/abs/1809.10790 by NVIDIA uses photorealistic images and something we call domain randomization to efficiently bridge the sim2real gap for robot grasping.

A core issue seems to be that the better the simulation, the harder it is to generate (compute wise) but simple tweaks to the generative process also go a long way.

It seems the tide is turning for sim2real. This article "Robots Must Be Ephemeralized" goes into some details, recently posted on HN: https://news.ycombinator.com/item?id=28605633

Reply on news.ycombinator.com