Total Relighting: Learning to Relight Portraits for Background Replacement | Hacker News Reader

Total Relighting: Learning to Relight Portraits for Background Replacement

augmentedperception.github.io

67 pointsHard_Space7 comments

Interesting - as I understand it, one of the big tech innovations from Disney's The Mandalorian was LED projected virtual sets that solved the lighting issues from green screens (see ~2:10 mark of this video: https://www.youtube.com/watch?v=Ufp8weYYDE8), making VFX scenes cheaper and faster to produce in exchange for higher up front costs. It had the added benefit of giving actors more to work off of.

I wonder if this technique will swing the cost pendulum back in favor of traditional green screens?

Incredible work. Coming from a POV of a person who has to wrangle a lot of VFX heavy creative, it's great to see the technical walkthroughs like this, especially the workflow overviews to see how it matches with our render process.

Is there code for this? There seems to be only a website in the repo:

https://github.com/augmentedperception/total_relighting

Don't expect code from this group, because their work usually goes into Google Pixel camera software.

What is the legal situation for implementing things from these kinds of papers? Am I allowed to use other's research papers to make features in my own products, or should I assume it's patented? I don't think copyright applies if I write the code myself, right? Or are neural net models copyrightable?

In this case you need specialized data from a lighting rig, but what about in general when you can use your own or open data to do the training, or in the cases when no machine learning is used?

Also of note -- building that lighting rig is actually not that hard if you have a spare walk-in closet and a bunch of WS2812 lights. You can programmatically turn on and off combinations of lights and automate the photography. But you'll probably want several subjects of multiple races and genders if you want it to work reliably though.

There's also this free dataset, although I haven't dug into it to see how good it is:

https://drive.google.com/drive/folders/10luekF8vV5vo2GFYPRCe...

( and the associated paper: https://zhhoper.github.io/dpr.html )

Are there any decent collections of human portrait Blender scenes? If so lighting would be easy to change in a pinch, and programmatically create a simulated dataset. A collection of images of this quality would be awesome:

https://blenderartists.org/t/felicia-blend/701475

It's probably patented (patent pending, because they take forever to get approved), copyright doesn't apply if you write the code and train the model yourself afaik

Reply on news.ycombinator.com