RealityScan Beta: Mobile Photogrammetry | Hacker News Reader

RealityScan Beta: Mobile Photogrammetry

capturingreality.com

88 pointsdagmx27 comments

That's the power of Epic Games' deep pockets.

Many people developing UE games need 3D assets, so Epic bought Quixel, a 3D asset store:

http://www.cgchannel.com/2019/11/epic-games-acquires-quixel/

Also people need to share their 3D models for colleagues, marketing, fans, etc. to look at. Quixel was using them internally anyway, so Epic bought Sketchfab:

https://www.epicgames.com/site/en-US/news/sketchfab-is-now-p...

And people want to create their own 3D assets, so Epic bought the company that Quixel was using for 3D scanning, called Capturing Reality:

https://www.epicgames.com/site/en-US/news/capturing-reality-...

And now you can go from reality to virtual twin to realtime 3D visualization without ever leaving Epic Games' servers.

People do not realize how big this is going to be - Epic is set up (and already is) to be huge in this space for cinema, TV, gaming, etc.

Anyone remember Epic's early DOS games, like Epic Pinball? How far they've come.

Exactly! At no point while playing ZZT in 1991 did I think "I bet this company is going to dominate the film industry 30 years from now"

> without ever leaving Epic Games' servers

That's probably a major driving point to the strategy after the Xbox Underground hacks.

Fun read of the tale: https://www.blackhatethicalhacking.com/articles/free-access/...

They got into Epic's network and that gave them access to many other companies.

Funny enough it wasn't the Apache simulator stuff that broke the camel's back, but the Gears of War 3 leak and Xbox One prototypes.

I remember the Gears 3 leak and at the time the conspiracies that Microsoft leaked it themselves:

https://kotaku.com/is-this-where-gears-of-war-3-was-leaked-f...

Scannings apps like this already exist though, like Scaniverse (recently acquired by Niantic) and Polycam. Both seem to output higher quality scans than RealityScan:

https://poly.cam/

https://scaniverse.com/

https://usdz.app

I'm a little torn on this. On the one hand, this is really impressive for on-the-fly photogrammetry. On the other hand though, this asset kinda just looks... bad? From a distance it's pretty passable, and in a relative vacuum of visual information it looks fine. However, the material itself is really sketchy. Whatever their default shader settings are makes the leather look toy-ish, caught somewhere between a reflective and diffuse material. The texturing is pretty low-quality (as expected), but the really annoying part is how it handles the ambient occlusion on the surface: it just looks like self-shadowing is baked into the texture.

It's a cool proof-of-concept, but I'm not quite sure who could use it in it's current state. Even by video game standards, these assets would probably look really out-of-place...

As a 3D artist who uses a lot of photoscans -- the scan quality looks fantastic to me! Photoscanned assets aren't usually meant to hold up up close, they're more intended for quickly building out a larger scene. Additionally, the material you see straight out of the app can be enhanced significantly with some common shader tricks like faking a bump/roughness map. Lastly, good lighting makes a world of difference when using photoscanned assets. Personally, I've just joined the beta and expect I'll be taking advantage of it heavily for several upcoming CGI projects.

All good! I'm not trying to detract from anyone's enjoyment/use of the product if they have one :p

I'm just a 3D hobbyist, and I don't really do a lot of photo scans. My thoughts on the matter are coming purely from a CG/asset generation perspective, if you're an artist with a little more agency with how you represent 3D objects then I bet you could work wonders with this (especially the geometry portion). Perhaps it was silly for me to expect these scans to rival hand-made assets, but my larger point was that we simply have a long ways to go before we reach parity with the two.

In the time that's passed, I actually did come up with a use for it though. It might be fun to use Blender's layer functionality to "trace" a 3D scan into a higher-res production model, just to get proportions right. I'll probably still hold off until this reaches general availability though.

I'd assume the model itself is the important part, the texture guessing is nice, but you'd probably want to apply your own PBR texture.

In my experience both go hand in hand. 3D scanned meshes aren't particularly detailed and texturing them is far from simple.

This looks bad to you? Admittedly I have little experience with 3D models. As a layperson this model looks incredible to me. If the real life results are anything close to that then photogrammetry is as good as solved, as far as my eyes are concerned.

Well, that's why I say I'm torn. I really do think this is cool, but when you compare it to hand-made assets the two would look starkly different. My biggest gripe is with the lighting: if you look at the creases of the couch, you can see that the lighting is literally "baked" in to the model. Shadows are rendered by the texture instead of the lighting engine, which means the models will appear static if you use them in a dynamically lit environment. Worse yet is the fact that these "shadows" can actually reflect light since the model has no reflection mapping or light mapping to tell the renderer that the shadows are rough as opposed to glossy. That gives the model the aforementioned "toy look" that I brought up earlier.

It's still quite cool, but asset capture has a long ways to go (even if everyone's results are this good).

This looks very interesting. Lately I've been playing a lot with the new photogrammetry API that Apple provides in Monterey on M1 (cloud instance) and it's working exceptionally well already compared to desktop based solutions to scan cars and humans. It fits my workflow of just using a phone, sending a video and getting a 3D model back (of course with some plumbing). I wonder how this compares and most importantly how the licensing model will look like? Is the processing done on the device or on a remote server farm? However, my congratulations on the release of the beta! I'm downloading it as we speak.

Are there any existing apps that use this API or another easy way for me to test this?

If you have a compatible mac there are several free apps like this one: https://apps.apple.com/us/app/3d-scanner-app/id1419913995

It's trivially easy to build one if you have a mac, they even have a demo project https://developer.apple.com/documentation/realitykit/creatin...

Anyone have suggestions for the easiest software to build a 3d environment (like in their demo) and drop in scanned models?

Or at least a path to accomplishing this with the lowest barriers to entry?

Maybe find some demos in the engine of your choice then import the model in?

About building the 3D model:

There is Regard3D, if by barriers to entry you mean sensor hardware: http://www.regard3d.org/index.php/documentation/introduction

If by barriers to entry you mean technical skills, there are SaaS solutions like Beholder: https://beholder.vision/

About embedding the demo in a web page, I have no idea.

Hi, I'm currently working on a project in that area. The main workflow in the system is take a bunch of pictures, SIFT & match them, then generate a sparse model of 2d features (image coords), 3d features (point cloud in relative coords) and camera positions. We then shard and re-combine different models to synthesize larger or more precise reconstructions. I'm building with clojurescript & Three.js[0] in the frontend, and can recommend the stack

Back to your comment I'd love to hear what you're looking to do and what you'd need, sounds like I may have explored the same problem space.

[0] https://threejs.org/

Hope iPhone use wasn't related to using Lidar aside from the images.

And I hope someone has finally found a way to enhance photogrammetry with depth sensor data because i'm pretty sure both combined has the potential to produce better results than either alone.

The photogrammetry api in macOS Monterey does this I believe

What types of features are you using for the SfM pipeline?

Algorithmic (SIFT) or ML based?

The chair looks awesome!

Can't wait :)

I digitized my flat with an m1 ipad and that looks already okay.

It's funny to see the flat layout after doing the 3d scan. An interesting way to persistent the flat memory.

Reply on news.ycombinator.com