Hacker News Reader

Comment by anon1m0us | Hacker News Reader

You'll notice the effect isn't as good at about 7 seconds into the movie where the green grass between the bride and her maids moves with the bride.

Why does the algorithm think that is part of her, rather than the background?

View on HN

Estimating the scene geometry from a single image is highly challenging and remains an unsolved problem. As such, you can find subtle artifacts like this in most of the results. The artifact that you are referring to stems from a geometry adjustment step in which we make sure that salient objects like humans or animals are on a plane in the 3D space. This requires image segmentation which is also highly challenging and may lead to parts of the background being assigned to the segmented foreground object and vice versa.

Could you not use something like the "magic wand" selection tool in image editing programs? Doesn't the dramatic color difference indicate it's not part of the same object?

I thought maybe it grabbed it because it was the same color as the bouquet she is holding, but that is entirely within the white of her dress.

There definitely is room for improvement. Our proposed framework uses Mask R-CNN for object segmentation. However, Mask R-CNN only as a resolution of 28x28 pixels for the object masks if I remember correctly (which is then resized to the actual resolution of the object in the image). So there are a lot of limitations from that alone.

if you haven't already seen it, https://github.com/dbolya/yolact generates very sharp instance masks very quickly, may be of interest

Those masks look great, thank you for the pointer!

yup! best of luck, love your research.

The two examples involving humans look terrible to me, the others look really nice. Probably some bias on my perception.

Yeah, we just are good at seeing artifacts in human renderings, it can easily look wrong or uncanny. In contrast, if a boulder is rendered geometrically wrong we usually do not notice it.

It's people moving up or down on the screen relative to one another in a way which their relative distances wouldn't dictate. Agreed that it gives me an uncanny-valley feeling (still super cool though).

Reply on news.ycombinator.com