Generating Faces with Deconvolution Networks | Hacker News Reader

Generating Faces with Deconvolution Networks

zo7.github.io

172 pointszo720 comments

I wonder if you could use this to do extremely low bandwidth video conferencing. Send a network configuration at connect time (relatively expensive) but for each frame, send the convolution network parameters, and deconvolve at the receiving side. Presumably this would result in far less data over time than sending the whole image. Sort of face-specific compression.

Ha, very similar to the original intention of the vocoder!

That Kraftwerk/Daft Punk voice, the original idea developed by Bell Labs was to come with ways to compress voice signals maximally by representing them only by the envelope necessary to modulate a bank of oscillators. Like any codec/compression scheme, the basic idea is to come up with a simple set of basis vectors that can represent the same data in a minimal fashion as long as it can be reconstructed on the other end.

http://www.bell-labs.com/newsroom/publications/290623/

"The method specifies the speech signal in terms of its short-time amplitude and phase spectra."

In the case of machine learning, the idea is simply to come up with those basis vectors empirically and automatically instead of reducing them to a mathematical model such as "set of sinusoidal waves".

Possibly, but one thing the article doesn't address is the computational costs of doing the convolution. Because you have to store an intermediate representation of the image for every convolution kernel that you use, you start to eat up memory in the GPU. I think processing should be quick enough that you could generate animations in real-time, however.

I'm sure I've read sci fi that uses the idea - they have extremely limited bandwidth so they make up for it with aggressive compression and reconstruction that more or less reduces people to a handful of parameters and uses those to animate an avatar. It relies on computation being a LOT cheaper than bandwidth.

Vinge's A Fire Upon the Deep features this, with some hilarity and a bit of foreshadowing.

I think Verner Vinge's A Fire Upon the Deep describes a teleconferencing system based on avatar models combined with the receiving side's video image history of the sender. It really had nothing to do with the story, other than a technical detour about ship-to-ship communication. The novel also had a galaxy-spanning Usenet-like system that was part of the story. :)

Why bother with this instead of sending a 3d model and just sending the position of a few control points/bone animation.

What the OP described would be extremely low bandwidth -- you only need three parameters to describe what to draw per keyframe (maybe one or two more if you want to describe mixtures of emotions or identities or more complex things), whereas you'd need more to describe 3D animation.

It's not a huge benefit over what you're suggesting (realistically, when are your bandwidth constraints ever that strict?) but it's an interesting idea.

Thanks for sharing the code! I've been looking to explore deep learning a bit more with Python, this is a great place to dive in head-first.

Yeah, Keras especially has a really intuitive and easy-to-use api, so definitely start there if you want to do more with deep learning.

It's almost too easy, really...

Very A Scanner Darkly -esque. Let's give it up for the vague blur!

Exactly what I was reminded of when I saw this. Very cool stuff.

Some of the "illegal" inputs results remind me of Francis Bacon studies.

Yikes, I wouldn't want to stare at those images too long. Very cool stuff, though. Could have an impact on the 3D animation industry.

There are companies trying to use this type of thing for game content generation. I recall some guy at tech crunch disrupt won an award last year (two years ago?) for something along these lines?

Just happened to have read some fiction set in WH40K universe and... now I know what exactly a mutant possessed by Chaos looks like. This makes a perfect illustration.

Has there been work done generating 3D models of faces? Like Mesh files that can be loaded into Open GL.

This seems useful for animation.

My immediate reaction too. Take some animated character and generate emotions at will...

Soon we'll be able to make full scale live action movies with no actors.

Reply on news.ycombinator.com