Joint 3D Face Reconstruction | Hacker News Reader

Joint 3D Face Reconstruction

github.com

124 pointstonylemesmer24 comments

The shortened HN title here is nonsensical.

In this context, "joint" means "together" or "at the same time". The work's title, "Joint 3D Face Reconstruction and Dense Alignment", essentially means "Face Reconstruction with simultaneous Dense Alignment". The HN title should mention both parts, or remove the word "joint".

yep - my mistake.

University of Basel has some nice project for 3d Modeling. They also have a scala project for 3d face model construction using Gaussian Processes. All the recent 3d modeling papers use Basel Face Model.

https://www.futurelearn.com/courses/statistical-shape-modell...

http://gravis.dmi.unibas.ch/PMM/

https://github.com/unibas-gravis/scalismo-faces

New meme? https://github.com/YadiraF/PRNet/raw/master/Docs/images/reco...

I just realized what this would be great for: Avatars in VR games. Snap a photo or video of your face and have it put straight into your character complete with animations. Excellent.

Check out an iPhone app called AffdexMe. It turns a photo into an animated 3d avatar in a few seconds.

I work in AR/VR and we are evaluating whether the iphone can be used for decent facial capture.

Isn’t the answer already yes? https://m.youtube.com/watch?v=i51CizUXd7A

I’d love to know what it’s short comings are vs other solutions

But we all know what this'll be used for.

Spoiler: pr0n

Wonder why it's written in Python 2.7 and not 3?

In my experience with scientists, they just use the tools that know how to use and have available. So maybe they just got the set-up for 2.7 on their personal/work computers or maybe some tool/library is using 2.7?

Nice project! I've been dabbling in DL for about a year (skimmed Stanford CNN course, Silver's RNN course, Andrew Ng's old ML course, etc.). While I can recreate basic stuff like MNIST, I don't feel like I can attack a problem like Pose estimation yet. How long does it take? Is it about diving deep into a single problem? Is it worth doing a nanodegree to shore up gaps?

I wonder if this can be extended for the whole body, so as to produce a 3D scan of the skin, and measure the evolution of moles and skin issues.

>and measure the evolution of moles and skin issues

be still my heart!

Only white people used as examples. Might be useful to literally diversify your dataset.

The arxiv paper [0] shows some further examples, which include some non-white faces. Your point is a fair one in the case of this application, since the goal is to do 3D reconstruction of the face. The project's YouTube video [1] shows one example (around 0:44) of the reconstruction run on test input of a black woman, and the reconstruction has some troubles modeling the wider nose common in black populations. More varied data (or an ensemble method that attempts coarse racial categorization and then further refining ethnicity-specific face models) might get better results.

[0] https://arxiv.org/pdf/1803.07835.pdf

[1] https://www.youtube.com/watch?v=tXTgLSyIha8&feature=youtu.be

This is somewhat ironic given that all five of the paper’s authors are East Asian.

Fei Fei Li recently pointed out in a talk that if you search google images for “grandma” you get very biased results. They’re all white grandmothers!

So it may be that the data sources the authors had are biased too. In any case this is an issue that should be addressed.

That made me wonder about what would happen if google started grouping concepts (eg words for grandmother in other languages will almost certainly give you differing results) rather than words...

Possibly (probably) the cognitive load on people might lead to bad A/B test results, but it would be a curious thing to explore - cultural bias would get a bit of a beating I'd imagine.

> In any case this is an issue that should be addressed

Why is that? This is just a demonstration of a new idea with existing technology, not some ready to deploy app for 2 billion people.

Absolutely. Even if one doesn't care about quality or inclusion, making this mistake can lead to big PR fails (and apparently lasting tech struggles): https://www.wired.com/story/when-it-comes-to-gorillas-google...

They're also all humans. How diverse they go depends on the application people put it to. It's surely enough to prove the concept with one race and one species.

Good READMEs are important and this is a great example of a good README. The gif at the top alone instantly tells you a lot about what the project is about.

Looking through the repo, anyone help me find where they put the vertices data that maps to image set?

Have at it, deep fake geeks...

Reply on news.ycombinator.com