We actually use both. The data used to train the neural networks is all generated through a scripted policy simulation. We then use a single human demonstration in VR to show the robot how to carry out a particular task. VR lets a human do a demonstration quite naturally, and would also be an effective way to collect training data for more complex tasks, where we’re not be able to create a scripted policy.

It's demonstrating one-shot learning. If you want the robot to do something that you can do, just doing it while the robot watches is a lot easier than building simulations.

It's not easy to get a million training examples, if you can learn something useful from just one example, that's great.

You can generate lots and lots of simulated data. DL needs huge amounts of training data, and CG is basically a solved problem.

Actually, for me the most interesting thing is that training on synthetic images transfers so well to real camera frames. That means that I might actually be able to do some cutting edge stuff without a huge curated dataset.

vs simulation, it lets you get trajectories from the way people actually move without having to make insanely complicated models. vs. precollected training data, maybe the recording apparatus is less expensive/complex?