Apple's Cubify Anything: Scaling Indoor 3D Object Detection | Hacker News Reader

Apple's Cubify Anything: Scaling Indoor 3D Object Detection

github.com

188 pointsTycho8727 comments

In case anyone is interested in rendering USDZ scans in Three.js, I created a demo: https://usdz-threejs-viewer.vercel.app/

Very nice and smooth! Do you have source for your demo?

There is one in the Three.JS example suite with source:

https://threejs.org/examples/webgl_loader_usdz.html

Fyi, looks mighty black and white on a pixel 6a.

They overcomplicate by using 3-4 different (sub) license in one project:

in README:

Licenses - The sample code is released under Apple Sample Code License.

- The data is released under CC-by-NC-ND.

- The models are released under Apple ML Research Model Terms of Use.

Acknowledgements

- We use and acknowledge contributions from multiple open-source projects in ACKNOWLEDGEMENTS."

in repo file LICENSE LICENSE_MODEL

why making it so confusing and elaborate? Its so useless to even use by 3rd party devs for making apps and releasing on their platform. So then just make it one license with the most strict restrictions you can make AGPL and/or CC-by-NC-ND .

They could have transformed it from insane to sublime by slapping a highly restrictive license on the readme itself. Seriously missed opportunity.

It complicated, but it's not overcomplicated. CC is not adequate for code and I belive that none of the code is GPL so your suggestion regarding AGPL is strange.

Why isn't CC-by-NC-ND adequate for code? Kinda makes sense IMO and the summary looks useful?

> CC-BY-NC-ND is a type of Creative Commons license that allows others to use a work non-commercially, but they cannot modify it or create derivative works. This means the original work can be shared, but it must remain unchanged and cannot be used for commercial purposes.

Notwithstanding it's only applied to the data in this case, it sure looks like a useful license for code.

> Why isn't CC-by-NC-ND adequate for code? Kinda makes sense IMO and the summary looks useful?

Because the Creative Commons folks themselves say it’s not because it doesn’t cover a number of software specific edge cases.

I'm curious what those are. I've never heard that before.

https://creativecommons.org/faq/#can-i-apply-a-creative-comm...

TLDR CC licenses don't cover source code distribution, patent concerns and general incompatibility with other software licenses.

Looks promising but the license, Attribution-NonCommercial-NoDerivatives is pretty limiting..

That’s just for the data, isn’t it, the code is Apple Sample Code License which I seem to recall is an MIT type license

"models are released under Apple ML Research Model Terms of Use."

Surprised this isn’t in coreML. Seems useful for the Vision Pro or something

Might see it at WWDC this year?

I keep meaning to get back to my suite of equirectangular image functions - viewers, editors, authoring etc. and this reminded me to resurrect the Viewer.

https://equinaut.surge.sh/?eqr=https://raw.githubusercontent...

Not quite right I think because the source image issn't 2x1 aspect ratio.

They can look really nice: both in the real world - https://equinaut.surge.sh/?eqr=https://upload.wikimedia.org/...

the virtual world: https://equinaut.surge.sh/?eqr=https://live.staticflickr.com...

I really want an app I can scan my whole house with the camera/lidar combo on my phone, and export it into Blender, where I can then rearrange furniture and stuff. Apps like Scaniverse get you pretty close, but everything is one mesh, would be great to be able to slide the couch around the space without having the manually cut it out of the mesh.

Yeah, how hard can it be :)

These days? Doesn't seem that hard honestly.

The accuracy of the results don't seem that great. For example, looking at the pictures on the wall in their sample, or the beams in the ceiling.

It's possible it's some artifact of the processing resolution, but I think most people that have worked with NNs for AR input will be surprised that this is not considered disappointing.

> The accuracy of the results don't seem that great. For example, looking at the pictures on the wall in their sample, or the beams in the ceiling.

Do you mean the accuracy of the classification or the precision of the lidar scans?

In my experience the lidar precision on the iPhones is decent but not great, so the texture mapping can look a bit off at times.

I'd love to have these bounding boxes on my scans though.

I mean the accuracy with which it's locating the bounds. What is extra curious is it obviously supports rotated cubes, yet it often doesn't use them when it should, leading to overstating the bounds, as if it's over enthusiastically trying to put things aligned to some inferred axis.

This is obviously an attempt at the general case to apply cubes to anything, but what is disappointing is the performance on boxy objects is lower than I've seen on private NNs used for AR and CV for years (ironically enough on iPads), using just rgb and no depth.

I half think the exercise here was to establish if transformers were the way to go for this, and on the strength of that the answer would be probably not.

Is this so your smart speaker can better report whats in your house back to apple?

Will it work on a picture of a Power Mac G4 Cube[0]? Whenever I see "cube" and "apple" together (which, in fairness, is rare), I think of the Cube.

[0] https://en.wikipedia.org/wiki/Power_Mac_G4_Cube

Reply on news.ycombinator.com