Video-to-Video Synthesis [video] | Hacker News Reader

Video-to-Video Synthesis [video]

youtube.com

62 pointsnakami12 comments

It's nice but I wonder why there's still nobody that tried to create a spatially coherent 3D space from the input before re-synthesizing the output. It's definitely possible and lots of work on this was done in late 2017. There should be more papers on creating and rendering 3D representations by now.

I vaguely remember a demo showing a 3D model of a winter trail, which was recreated (and textured) from a sequence of photos. It was done with Photosynth, I believe: https://en.wikipedia.org/wiki/Photosynth

I didn't find the winter trail video but stumbled upon some interesting links:

2018 3D Scanning: A Comprehensive Survey https://arxiv.org/abs/1801.08863

2012-2014ish BigSFM: Reconstructing the World from Internet Photos http://www.cs.cornell.edu/projects/bigsfm/

David J. Crandall https://www.cs.indiana.edu/~djcran/#research

Andrew Owens http://andrewowens.com/

Also check out his on/offscreen audio source separation work. It's pretty neat. (2018) Audio-Visual Scene Analysis with Self-Supervised Multisensory Features http://andrewowens.com/multisensory/

Noah Snavely http://www.cs.cornell.edu/~snavely/

(2018) MegaDepth: Learning Single-View Depth Prediction from Internet Photos http://www.cs.cornell.edu/projects/megadepth/ https://www.youtube.com/watch?v=tGbMWAFMMBQ

With live demo page :) http://megadepthdemo.pythonanywhere.com/

Does this mean that in the future we could literally be able to play FMV games?

Basically, an AI that has been trained on every digital film there is which in turn also generates unique permutations to further increase the training data....

Will be able to conjure up things like Breaking Bad Season 8: 'Lil Heisenberg and pretty much even helmet cam videos from war zones in Donetsk that you could control in real time....

Singularity indeed.

I want remakes of movies where characters can be swapped out from unrelated movies.

Basically, a simulation, with the plot as objectives.

Interesting but how do we know the examples are not cherry-picked?

Well, first, Nvidia has a good track record. This is not their first video-to-video paper, and they've also released impressive things like ProGAN; they release not just the source code, but trained models as well. People have trained their own models (slowly) and poked at the released models. The average result might not be quite as good, but they'll be close.

Second, in this case, they've also released source code and the trained models. The trained models aren't in the Github repo, but hosted on Google Drive and they provide scripts for downloading them: https://github.com/NVIDIA/vid2vid/tree/master/scripts

If you are doubtful, simply install Pytorch, download the models, and give the video generation a try. (You don't need a GPU, even, since for generating some samples once, a CPU will be acceptably fast.)

Wait, this algorithm works at real-time on consumer grade hardware ? I am impressed.

I thought a forward pass for the model would be a lot more expensive than that.

I never said it was realtime. I said it was acceptably fast on CPU ie. it'd take a few minutes/hours to generate some samples and convince yourself that they did not cherrypick a tiny handful of good quality samples out of a universe of lousy ones. I would very roughly guesstimate their 2000px model probably would take ~0.5s/frame on a GPU, so multiply by the usual >10x CPU slowdown, you can generate a 30fps 30s snippet in maybe an hour or so.

Wonder if this can be used to do video upscaling on steroids. Would love to watch classic TV shows in glorious 4K resolution.

What I want is classic movies in full 360 vr... I'm guessing this will exist in a few years, at the current pace of things.

What is the state-of-the-art in segmentation mapping in video?

Reply on news.ycombinator.com