Show HN: A web app to turn photos into 3D AR models. | Hacker News Reader

Show HN: A web app to turn photos into 3D AR models.

usdz.app

191 pointsschappim66 comments

Open articleView on HN

Hi I'm Marcus (one of the creators). Feel free to hit me up if you have any questions :-)

We are "very alpha" right now.

Edit: We have an API over at: https://usdz.app/api

What photogrammetry pipeline are you using?

It is a stupidly long AWS based pipeline w/ the photogrammetry part is in swift.

If it's not a secret, I'd love to read some design document about the backend behind this app. What's the blackbox that takes set of images as an input and produces a 3D model as an output?

We've very much built on the shoulders of giants:

  - Frontend & API (Rails + jumpstartrails.com)
  - Images --> USDZ (Swift Code on a Cloud hosted Mac)
  - USDZ --> GLTF (Swift + Custom Ruby)
  - CDN (AWS Cloudfront)

Lots of Glue:

  - AWS S3 / Cloudfront
  - AWS SQS

It uses the Object Capture API introduced in the latest version of macOS.

The API allows you to create 3D models from images with just a few lines of code.

https://developer.apple.com/augmented-reality/object-capture

If we're talking Swift frameworks, ModelIO (underrated) and ObjectCapture are certainly part of things.

He asked for the “blackbox that takes set of images as an input and produces a 3D model as an output.”

That black box is Object Capture API, which takes a folder of images and outputs an USDZ (or OBJ) file. Model I/O isn’t necessary.

Kudos to Apple for releasing a photogrammetry pipeline that allows web and app developers to build tools like this, with zero knowledge of 3D reconstruction required.

You're not wrong (althought we do use ModelIO a lot).

I just saw that you're from https://www.aboundlabs.com and metascan.ai Your scans look great! So how does your black box work?

I’ve just been trying out the photogrammetry session, it’s super simple to use. I think Apple’s photogrammetry software isn’t as advanced as other’s, but it is free and works well small scale.

I work in the same field, making a capture app for photogrammetry software

Yeah what Apple's has is a clear lead on the time required to build objects.

I think that the photogrammetry side of things, the "give me an object now" will shift towards mobile device camera apps.

It is only serverside now due to power requirements.

If they make the api available on ios it will be a game changer, i’m hoping they’ll make it available for the m1 chipped devices

The rumours say they're putting m1 level devices in the the headset. Again the power budget is the only issue!

Hi Marcus.

It would be nice to have some idea going in for what the requirements are. - I have to admit, I don't understand why I can't preview the 3D experiences right on my computer. - Likewise, when I try the QR code on my iPhone 6 ... well, I'm hardly surprised to not see anything in 3D, but it wouldn't hurt to have a little more detail than "Scan this QR code on your iPhone or iPad."

I suppose I could mention that I've done some game development and a little bit of VR/AR, but have never heard of the USDZ file format before.

Hi there!

USDZ is the format that Apple is backing for VR/AR delivery (developed by Pixar)[1]. We're betting that it is what is going to be used in any Apple headset.

Right now you can't easily view USDZs in the browser (the format is a moving target), however Apple is making strides to make it available with the proposed Model element[2].

We were doing some USDZ --> GLTF converstion (GLTF being a format that will work in the browser), but it is jankey (technical term).

I have not tried an iPhone 6 yet. I will have to pry it from my children's hands and remove a layer of cheese and excretions before I do.

I've made a video on how to do this via the browser here[3].

TBH, the experience a heap better in the iOS App (currently in testflight) [4].

[1] https://graphics.pixar.com/usd/release/wp_usdz.html (seem to have cert issues) [2] https://github.com/WebKit/explainers/tree/main/model [3] https://usdz.app/screencasts/getting-started [4] https://testflight.apple.com/join/VteIxFd7

Oh, I should also mention, HEICs are your friend (as they have depth, and accelerometer data embedded in the file format).

Anyway to take a model that was created and get it into CAD / modeling software? I don't know much about VR/AR but I know it is a pain in the bum to replicate an object / create a model for 3d printing.

Thank you.

We allow you to download as USDZs (and sometimes do conversions to GLTFs).

You can import both of these into Blender and export as Obj (and I'm pretty sure STLs).

You may want to use our API to do the Images ---> USDZ conversion[1].

[1] https://usdz.app/api

Excellent. Thank you for the reply.

This is really great, thanks for doing this!

Hey very cool stuff, one piece of feedback it would be good to be able to delete models on the beta iPhone app, or if we can it’s not particularly obvious how to do so.

You're right! Right now it is not possible (via the phone). It should be possible via the API or https://usdz.app

Pretty cool!

How are you handling spam or other kinds of content moderation? Is it easier because users don’t upload a regular text/image and need to create something custom?

There hasn't been anything humans can't handle.

If it gets out of hand, we'll look at something similar to Akismet, or an ML based solution.

Right now we're hand picking experiences to make it to the front page.

At early stage you are not going to get spam anyway. Once it grows (and i hope it does) you will get fare share of spam. As you said at that point you could check solutions like Akismet, OOPSpam.

Thanks for answering! Hand picking sounds like a really good approach, simple and effective.

So funny you should mention this, yesterday we had our "first porn" moment.

Thankfully if one throws a bunch of random images at the API and trys to make a model, the algorithm just fails.

However we're thinking it might be fun to do something w/ NSFW image detection and display a silly model...

Hey have you guys considered licensing an API?

We don’t think is a good long term business model (with the obvious path of functionality moving to the client).

Smart. Looking forward to this being an on device capability.

If you don't have an iphone, you can view he model by downloading them in the "options" drop-down menu to obtain the usdz file, which you can then use either with Pixar open-source suite https://github.com/PixarAnimationStudios/USD or other software that support the format.

Or you can use an online file converter like https://products.aspose.app/3d/en/conversion/usd-to-obj and see the result in Meshlab.

Direct in-browser visualization is possible but https://products.aspose.app/3d/fr/viewer worked but the scale was wrong and prevented to zoom enough.

You can also view some high quality scan in the Apple's demo https://developer.apple.com/augmented-reality/quick-look/

(Edit: The ones with the "Made with object Capture" label which this service is a front-end for https://developer.apple.com/augmented-reality/object-capture... ).

Yup, good tip. We gotta make it nicer for Android.

We have a USDZ to GLTF (Chrome/Android) converter too. We should make the gltf more readily available.

Very cool, reminds me a bit of the photogrammetry software Autodesk has. Always thought it could be more "consumerized" so that a series of photos can easily be made into 3D models.

This was also a nice throwback for me to some drone photographs I've taken a while back at Berkeley stitched together into a 3d model: https://twitter.com/zuhayeer/status/933752635726880768?s=20

Thanks for your kind words! Ultimately I think this is going to be part of the camera apps or some built-in app. It is just a function of power.

Very Interesting project! esp. when compared to the swathe of junky or overly complex photogrammetry tools out there from a few years back (but mostly still polluting the app store)

I've only recently been trying out poly.cam which is probably the best example of a user friendly workflow ... but it's knowingly priced. Have you guys got a pricing model in mind? Also do you have any documentation around best practices/results, etc?

Anyway sounds like things are going to pick up again quickly with apple's new APIs. Good luck - exciting times ahead!

Thanks for the tip!

I'm making a "how-to" video now. I've yet to do the voiceover, but here are the raw slides: https://files.littlebird.com.au/take-great-shots-cXH2CM.m4v

The TL;DR is:

  - Avoid thin objects
  - Avoid reflective objects
  - Avoid transparent objects
  - Avoid objects that deform
  - Hardshadows are not good, use soft lighting
  - Your photos should have a crisp focus and be well lit (across the object) 
  - Things work best when the background has a solid colour
  - Make the object take up the full frame (more data for the algorithm)
  - Make sure to get shots from all sides

As far as best practices, Apple has some documentation on how to use their photogrammetry pipeline:

https://developer.apple.com/documentation/realitykit/capturi...

Hey Nobbis, I've just requested an invite to your SDK competitor to Apple and others. What you've done looks great and I have a few questions:

  - Does your solution have the same photogrammetry requirements as the Apple/others?

  - Is there a reason you're only supporting iOS at this time? Is that because you're using an iOS framework, or because you're relying on a file format (say HEIC)?

  - How long until you guys open up the API?

Keep up the good work!

Is there something like neutral lighting where a texture can be photographed and then the engine's artificial lighting makes it look realistic post lighting?

yes

Where can I read more about this? What is the terminology for it?

Generally for photogrammetry you want to aim for more diffuse lighting. Avoid direct light sources as they will cast shadows onto your subject that get baked into the texture. Cloudy days or light diffuser umbrellas are helpful.

https://sketchfab.com/blogs/community/lighting-in-photogramm...

There are methods for de-lighting, but it’s better to just capture without if possible.

Or if you want to get fancy you can build a capture rig to create intentional shadows and process them into texture normals with something like Substance.

https://store.substance3d.com/blog/your-smartphone-material-...

Further to this, I've made a video on how to get the best shots:

https://usdz.app/screencasts/how-to-take-great-ar-object-cap...

I think I've seen it at top 10 vision papers 2021[0]

[0]https://www.youtube.com/watch?v=8bHzytU1SeM

This looks great, congrats! I’ve tried a number of photogrammetry software and they typically had challenges. One question: are you able to determine the size of the object?

Thank you for making this! This is exactly the easy solution I was looking for a few days ago for a side project.

Looks Excellent! Going to try this out tonight in Final Cut Pro with the mTracker 3D plugin.

Nice! One tip, if you have Motion (which there is a good chance being a FCP guy, you do), you can import the USDZ file into Motion.

You can download the high quality version of a model here[1].

[1] https://files.littlebird.com.au/Shared-Image-2022-01-02-18-1...

I do - thank you very much! Had no idea when I woke up this morning that I would be scanning my four year-old’s dinosaurs into it.

Nice! I have been "borrowing" the kid's toys too. With enough photos you can get some great shots (such as this one)[1]. Looks better as a USDZ, than a GLTF.

[1] https://usdz.app/experiences/284-dragon

This looks great. Unfortunately the app is not working for me on an iPhone SE 2020. I am getting an ‘Error accessing your camera. Please check your permissions’.

The permission to allow Hologrid to access my camera is correct in settings.

The dog is oriented in 3d somehow in a “_/_“ way, instead of lying on the floor.

Side question: how the site has access to my camera, when it’s set to disabled in my ios-safari settings? Settings - Safari - Camera - Deny.

Could be the difference between file upload (via camera) and media device access?

Testing it… I’m not sure. I tap on the 3d cube logo in the picture of an object, and then “camera screen” pops up, but it’s not the camera app, it’s safari, I see that in the apps list (by double clicking the home button). Nothing in downloads either. Very strange.

Would love to see some examples of how to go from this into something like A-frame - or even a guide on how it might work?

Your wish is my command.

  - This is one I'm building for mobile (work in progress) [1]
  - This is one for desktop [2]

[1] https://files.littlebird.com.au/mobile-how-to-wip-BDikJ9.m4v

[2] https://usdz.app/screencasts/getting-started

testflight app doesn't seem to be able to access my camera (iPhone SE2) despite having permissions

Thanks for the report Mitch. We'll have to look into this.

Having the same issue, using the SE 2nd too.

Does this have android support?

It will. We've been building the mobile app using a cross platform framework (Capacitor) with the view of making it Android compatible. Capacitor does the job, but we've had to do some native code to fill holes.

If you're a dev, you can also use the API: https://usdz.app/api

You could probably whip up a quick script that runs on Android. My first go of doing mobile (for enlisting my co-founder) was to use Apple Shortcuts w/ the API.

I can downvote comments but not stories. Is there a karma threshold for downvoting stories?

EDIT:Ugh, sorry about this comment. Looks like I somehow posted it under the wrong story. No clue how this happened. Looks like I'm unable to delete it as the deletion time window has expired.

Stories can’t be downvoted.

If you think it isn’t appropriate for the site, there’s a flag link. (I can’t imagine that applies here.)

How many photos does it take?

I think an API for this would be ridiculously cool.

You need at least 20-ish, 70-200 make amazing objects.

There is a free API here: https://usdz.app/api

Reply on news.ycombinator.com