SceneScript, a novel approach for 3D scene reconstruction
ai.meta.com
ai.meta.com
The semantic understanding of the objects around you (without upload to a server).
A detailed set of actual meshes of the objects in your living and working spaces.
So much can be built on this key building block, to actually help people with their daily lives, low vision people are only one group that will see massive benefits.
I also think this technique could and should be extended generally into computer vision applications, understanding semantic meaning, volumes and masses via the often highly structured environments around us.
Starting from a principled position of meaning, volumes and meshes and then training to assign gaussian splats to those meshes seems like something clever people could solve a lot of problems with.
> without upload to a server
Doesn't matter. If the app has to render them then it can still send them to the server. If the app does not render directly it can still infer them by doing collision testing against them. If it can't even check for collisions then there isn't much point in having the data in the first place.
It gets worse. The more you want AR/MR to understand the scene the more fine grained data you need. The camera sees things like: the underwear you left on the floor, the medicines/pills on your table, the posters on your wall, the bank account browser tab still on your computer screen, the adult toys you forgot to put away, your undressed body, your dressed body, your political baseball cap or t-shirt, your naked roommate/spouse/kids who just walked into the room.
I don't see how this can be solved in a trust-less way. In other words, you'll have to trust each and every individual AR/MR app you use, that they aren't spying on you. The platform/device/OS itself can not prevent this and still allow the majority of the AR/MR uses that people imagine.
You can't be probusiness and pro-privacy at the same time though, you need to limit businesses in order to protect the people.
While this approach doesn’t address privacy completely, it avoids certain elements of our life that are considered really private.
I think half of the point of using the simulated dataset to train the model was to safeguard the above mentioned kind of privacy (with the other half being lack of real-world dataset).
But you would defo have to trust the OS, and in this case Meta…
I sometimes feel like this may not work without language helping it along
"where are my keys?"
It should be able to respond -
"Here on the dining room table" with an astar path drawn to them in the headset.
It's a stream of tokens that you can almost just reverse search on to find the last instance of "keys" in it (thats a simplification but...).
My feeling is the better approach is to train a generalized segmentation model, derive multi-parameter representations of those segmented ‘things’, derive embeddings from those things for compression purposes, then allow users to ‘tag’ or ‘label’ or ‘describe’ those things as a service to other users. Then we could choose whose language about the world we ‘subscribe’ to, and anyone could attach any digital payload to any real thing/person/place without anyone else’s help or permission.
Walk around your home and notice [almost] everything already has a name for it as it comes from some store (unique handmade items are an outlier). Let's just label things we all agree they are called (or give them multiple labels).
Believing there is one map from the real world to the world of symbols is solipsism.
With a system like the one in the video you walk through the old place, and walk through the new place, and output for example 2 Sweet Home 3d plan's.
No need to have 3d glasses for that.
> That understanding would let AR glasses tailor content to you and your individual context, like seamlessly blending a digital overlay with your physical space
Horrifying, if it ever takes off in the way that "Meta" wants it to. Take the modern smartphone disease and amplify it a few orders of magnitude.
> While LLMs rely on vast amounts of training data that typically comes from a range of publicly available text sources on the web, no such repository of information yet exists for physical spaces at the scale needed for training an end-to-end model.
Considering the tremendous scale in question, the "yet" is also terrifying. The only way to collect such a dataset would be if some large fraction of people were wearing always-on, network connected cameras. Meta clearly sees this future and wants it; the kind of future where everyone wears a Meta headset, all the time, and has their experience of reality mediated through it - optimized to maximize shareholder value of course, under the watchful eye of The Algorithm. A panopticon AI shoggoth.
Meta is an AI paperclip maximizer, and it is getting uncomfortably close to the "RELEASE THE HYPNODRONES" stage.
That made my day. I do agree, it is creepy but its what theyre pushing.
As egregious as this is, it still doesn't strike me as nakedly insulting as Google's refusal to let you turn on the (entirely passive!) GPS receiver on your phone without uploading your precise location to their servers, for similar data collection purposes.
I'd love to see this sort of thing be made illegal, but the only regulatory domain with an Overton window even close to that is the EU, and it's such a minefield of potential gray areas that I doubt it would get off the ground even there.
You must have missed the part where they created a synthetic dataset of 100000 environments, trained the network on that, and released it to the public?
Not saying I’m not worried about their plans too, but that is clearly addressed in the text.
* How uncomfortable the headsets are to wear for long periods if time
* How useless they are (what are they solution for?)
* How ridiculous they look
* How expensive they are
* How expensive it is to produce content for
After they solve these issues then I will worry about the dystopia we are in. VR headsets may look cool and futuristic in the movies but in reality they are just screens you wear on your head and that's it.
And they won't be useless! That's the worst part. They will provide compelling value, like smartphones, and as such will be impossible to boycott. At first the value will be simple, obvious utility, like route overlays. Later it will be value within the system, much like how the main utility of smartphones now is to run the mandatory apps to participate in aspects of society that previously did not require a smartphone. And of course they will be addictive.
Here is a short sci-fi film depicting what I fear: https://www.youtube.com/watch?v=YJg02ivYzSs
Now as far as making them more useful I suppose *if* they can solve that first problem plus make them match the form of IMAX glasses (preferably smaller). Then yes they could be useful (providing there's good enough applications).
But don't cause the user to get sick feature is not 'a nice to have' you can't expect it to spread like the smartphone if people can't use it for long periods of time.
Will Meta solve that problem? Doubtful. Will they provide good enough applications to make AR worth using once they do? I wouldn't bet on it.
Just because a company has an incentive for something to be successful doesn't mean they can pull off. Just look at metaverse.
Wait, Snowcrash was horror for you? I thought the whole point of post-cyberpunk was to not assume all tech would be dystopian.