High-Resolution 3D Human Digitization | Hacker News Reader

High-Resolution 3D Human Digitization

shunsukesaito.github.io

129 pointshliyan30 comments

I've tried many of these papers, and they don't work outside their datasets.

Suppose you tried this on your own camera or maybe in a different lighting: it would totally break this.

Paper definitely seems like an improvement, but I guess no one should get excited to use this in experimentation/production.

For what it’s worth, this project did get an update about a month ago. I can think of some cool uses for this if it works.

https://colab.research.google.com/drive/1GFSsqP2BWz4gtq0e-nk...

Interesting cool applications from when I was a PhD student [1,2].

[1] http://www.eurecom.fr/en/publication/3189?&theme=mobieurecom

[2] http://www.eurecom.fr/en/publication/3247/copyright?popup=1

I've tried a number of ML projects like this too, and sometimes they do work – Bodypix is one example (though not precisely the same thing).

That being said, its usually pretty difficult to get code from papers to build/run successfully due to dependencies (e.g. depending on a specific version of Python and OpenCV 2 and requiring CUDA support)

Could anyone explain why version numbers make ML stuff so brittle? Why is the CUDA version supposed to match? Why would a newer version of python (3.7 vs 3.6) would ever break anything?

TensorFlow is a crapshoot as far as I understand, constantly changing making newer versions incompatible; but why do other libraries break backwards compatibility without a major version bump?

Because software is hard.

You might be relying on a function that only exists in 3.7 and not in 3.6, code written in 3.6 would work but new code using 3.7 features won’t be backwards compatible. With compiled code the errors are usually very hard for “more used to scripts“ people to decode. You get stuff like missing symbols in the linker phase.

ML projects usually have a lot of libraries so you also get in the transient dependencies breaking quite often...

Yeah, managing dependencies is tricky. I've been super excited about Nix and Guix lately for that reason; if you have a single Nix/Guix revision and a list of packages, you have all the information you need to build exactly the same package tree. (With bit-for-bit reproducibility where possible, no less!)

Some language-specific package managers can do similar things, but you really only get reproducibility for the whole system with a general-purpose package manager. Poetry gets you pretty far within the Python ecosystem, but if you need a specific version of Python/specific native libraries/etc... it doesn't get you all the way there.

Because dependency management for anything C/C++/Python related has often been a massive cluster* * * *. Rust's npm style package management is probably the greatest innovation to have hit low level programming in a long time. Not to mention anything Nvidia related is often closed sourced so you are programming against a black box API by a vendor infamous for their bad driver quality on non-Windows systems.

I tried to see what would happen when I put in some stock photos of people into the Colab notebook linked from the repo's README. Some worked better than others. Some totally didn't resolve well at all. Overall I think it's interesting, but there are definitely a lot of edge cases.

Some gifs of the test results: https://github.com/cardboard-q/pifuhd_demo_model_test

"yoga" example is quite disturbing.

CVPR 2020 (starting today!) has a lot of papers on the topic for interested people.

Here is PiFuHD: https://arxiv.org/abs/2004.00452

The next thing might be mapping from that static 3D model to some kind of rigged model that can be animated. Then you would not only need to place the bones and joints but also separate the clothing from the body. Extremely hard but DL has been able to pull off some incredible stuff.

Wow. It's such "high resolution" that the distorted triangular polygons of the untextured models are clearly distinguishable and painfully obvious on my phone.

I guess my idea of "high resolution" differs from what the rest of the world describes as such.

It's interesting how the legs are often an unequal length in the reconstruction when the person is walking, due to some incorrect interpretation of the camera perspective.

I would have guessed this wouldn't be a problem given that there are multiple photos of the same view, such that you can resolve the depth ambiguity from a single camera. I.e use feature detection to identify feet in > 1 views, and then use the two images to resolve the depth of the epipolar lines that lie on the optical axis and reconstruct the shoe in 3D space.

Straight up en route to Ready Player One. Guess this remote thing is happening for real for real

More like increased competition and computer generated avatars in the modeling and Instagram influencing industry.

I am interested in high resolution 3D mapping of skin surface e.g. for facial scar revision etc - what is the best technology available for this either research or commercial?

Pardon me for asking, but isn’t the name of the technique a sort of pun? 皮肤 means “skin” in Chinese and romanized as “pifu”

The last author and his company were/are involved in scandals about faking results on a large scale:

https://www.reddit.com/r/MachineLearning/comments/8zm4kl/d_l...

http://sadeghi.com/dr-iman-sadeghi-v-pinscreen-inc-et-al/

Thanks, I think this deserves an HN post of its own. Some of the things that were done to entice Sadeghi to join sends shivers down my spine.

Edit: I've been looking at the details on the rear view of some of the "Single-View Reconstruction" examples, and I'm starting to worry that this may actually not be reproducible.

I have followed this case since the beginning. I am surprised how the academic community (who's fully aware of this case) continues reviewing his work without questioning work ethics as though nothing happened. TLDR:

Dr. Iman Sadeghi is the man behind hair rendering tech for Disney and Dreamworks. He left his job at Google to join Hao Li's company Pinscreen (which by the way is funded by big names like Softbank).

When Sadeghi saw red flags inside the company, he raised the issue from within, and finally wanted out. While he was leaving one day, Li and his colleagues legit assaulted (violently) Sadeghi to give up his company laptop. This, by the way, is recorded on CCTV cameras and can be viewed online.

The fraud case is about falsifying results in their SIGGRAPH 2017 Technical Papers submission. They claimed to generate avatar hair shapes in their paper, and when their reviewer asked them to give results on many faces, the company hired artists for as much as $100 to generate them manually for the results. Of course, they later claimed to make it fully automatic AFTER publishing the paper, but that doesn't justify the stance that they published false results in one of the biggest computer graphics conference. Hell even at that public demo, they showed pre-cached avatars and claimed them to be generated real-time.

Reply on news.ycombinator.com