Hacker News Reader

Comment by nobbis | Hacker News Reader

Trust Google to look at their c. 1993 gouraud shaded, untextured renderings without shadows and conclude "let's use deep learning to make it look realistic!"

View on HN

ICCV 2017 this year had dozens of papers using GANs and other methods to make simulated images/videos look more realistic. The hunger for cheap usable data is real.

Can you provide some links? this sounds interesting ...

I spoke in hyperbole, there were dozens of papers using generative networks to create real looking simulated data which is the more common approach. You start with noise + some features you want in your image, then you turn that into a realistic image.

The notable exception is if you want to generate segmentation data (where you want labels for each pixel), in this case the set of features you want is essentially an image already.

ICCV 2017

* Photographic Image Synthesis with Cascaded Refinement Networks: https://arxiv.org/pdf/1707.09405.pdf

* Synthetic avatars: https://arxiv.org/pdf/1704.05693.pdf

* Synthesizing cell microscopy data: https://arxiv.org/pdf/1708.04692.pdf

* Material Editing Using a Physically Based Rendering Network: https://arxiv.org/pdf/1708.00106.pdf

* 3D-PRNN: Generating Shape Primitives With Recurrent Neural Networks:

* for more search 'generative' 'generating' 'simulating' 'synthesis' 'synthetic' on: http://iccv2017.thecvf.com/program/main_conference#schedule

Also worth mentioning:

CVPR 2017 best paper, this is the other exception, it's unique because it doesn't require _any_ labeled data

* Learning from Simulated and Unsupervised Images through Adversarial Training: https://arxiv.org/abs/1612.07828

This seems like an unreasonable approach given the quality of their graphics. They might as well train an GAN to replace the entire rendering pipeline if they are using just simple images! Train it to generate images directly from pose information. They really should focus on improving their rendering beyond a 1993 level if they want a robust solution.

(A note from elsewhere)

A reminder that a state of the art GPU renderer can produce photorealistic scenes at 4k resolution (about 8000000 pixels) at 120 fps (i.e. render the scene in less than 8ms).

In other words, this performance is completely inaccessible to current DNN architectures (and I suspect will never be as efficient). As another algorithmic illustration, we have already near-optimal algorithms for sorting large lists of random numbers. I guarantee a large neural network even trained for massive lengths of time cannot beat the sheer efficiency of traditional algorithms. It should approach the asymptotics of the optimal algorithm, but there's too much overhead in how DNNs are structured. A better task, I believe, is to use DNNs to write code that improves renderers, shaders, or sorting algorithms while maintaining image quality (in the case of computer graphics) or correctness (in the case of sorting algorithms).

---

In this case the problem could be coming up with a variety of good realistic textures. I'm sure Google could hire a few artists to do a fantastic job, but if you want overkill I guess a GAN could generate the textures (to be rendered by a conventional algorithm).

That is too indirect a problem. Neural networks will find answers to problems where there is a relatively direct hierarchical connection between input and output. Symbolic information would require different and very big architectures. Also doing that is still state-of-the-art and you don't try to solve 2 state-of-the-art problems at the same time.

Firstly, using the better rendering approach has been tried and failed. These networks exploit things that perfect rendering has but reality doesn't. For instance, in a rendered simulated environment it is perfectly reasonable to estimate exactly where an edge is and then follow that edge with zero margin with the robot. In reality you need a minimum margin if you expect it to work (or your object will be damaged), and a decent one.

Photorealistic rendering tries to emulate optics that are incredibly good, with no distortion, and the robots have cheap webcams with plastic lenses, with both consistent distortion near the edges and random distortions everywhere because they're webcams. Not supercheap ones, but this isn't a RED camera either. You could fix those lenses, but why would you ? It would be more valuable if it works with cheap lenses, and human and animal eyes don't really try to be distortion free. Rather they have the "software" behind it compensate.

Second, photorealistic rendering requires modeling accuracy, and material modeling on a scale we just can't do for real datasets. These datasets have millions of objects, and to make the robot more tolerant of the general weirdness of the real world, they are randomly perturbed, to turn millions of examples to billions. But those perturbed objects are not photorealistic anymore. Nobody has a toothbrush with a triangle in the bottom sticking out to one side.

Besides, keep in mind that the resolutions used here are quite small. It's not like they're trying to run a renderer at 4k resolution. Think 128x128, or 256x256. Not exactly a big issue.

Photorealism (or "better rendering") doesn't mean perfect imaging. You can model the radial distortion, rolling shutter, Bayer filtering of cheap webcams in 5 lines of code.

A better simulation is closer to reality, not closer to an ideal.

A "minimum margin" is needed anyway, even with perfect perception, because in general you don't have perfect actuation.

>They really should focus on improving their rendering beyond a 1993 level if they want a robust solution.

Recent work from openai suggests that the 1993 graphics with randomized textures may be good enough, and the push for realistic graphics may be entirely unnecessary to get a robot capable of generalizing it's learning to the real world.

I'm interested to know which simulation feature provides the most information. Sort of a random drop out of features on the scene graph. Include lighting vs. no lighting, color vs monochrome (brightness only), shadowing vs non, texture vs untextured, monocular vs binocular, etc...

You can test what simulation features are informative by measuring the resulting real world accuracy with the arm.

How simple can the simulation be? How informative is each feature?

It's the opposite. Simple rendering makes pose estimation harder, not easier - texture and shadows resolve depth ambiguity. So it's easier to show you've "closed the simulation-to-reality gap."

Reply on news.ycombinator.com