Face Swapping
kk.org
kk.org
The non-real looking photos have mostly to do with the mixups in gender/age/race, that we do not explicitly guard against currently.
Here's the project page describing things in more detail: http://www1.cs.columbia.edu/CAVE/projects/face_replace/
The video linked from that page explains the whole process from beginning to end in under 5 mins.
Given a face to replace and a database to find candidates from, the process is fully deterministic. The database itself can of course grow, or be switched out. This allows one to create different databases for different applications (e.g., one with synthetically rendered faces, one with stock photos, one with personal photos, etc.).
If this is right, then the program is agnostic to individual facial features, as in it's everything from the eyebrows to the lips, or nothing.
Also, if this is right, would there be privacy implications to the person whom the face is extracted from? If you are protecting the privacy of the person whose face is swapped out, then it makes sense that only their frame remains, but the hybrid face is very similar to the original face (second photo).
This is why I said earlier that one could create multiple databases. Since the replacement candidates might be recognizable, they should be from a set of people who've given their permission for their photos to be used (e.g., stock photo collections). Another option is to use computer graphics to render a large collection of synthetic faces. Modern techniques are good enough to generate fairly realistic faces.
Finally, note that if every face within some application has been replaced (e.g., if this were to be applied on all images on Google Streetview), it wouldn't even be a privacy issue for the replacement candidates -- since users would know that all faces are replaced, seeing a particular face only tells the user that that person was not there.
So... when are they going to roll out an (optional) auto-tagging feature?