Spatial Computing
hypercritical.co
hypercritical.co
Imagine that from the very beginning computers only had virtual reality headsets as the only interface. In the time of ENIAC it would be extremely huge grotesque steampunk helmet that you would put on but as the time goes by smaller and smaller headsets developed and eventually today we would arrive exactly where we are now with this Apple VR headset. And now, someone invents desktop PC monitor. It will be revolutionary. No longer you have to strap on headset and wear it for 8 hours when you're in the office. Instead of $4000 fragile gear you buy cheap $100 dollar flat panel with double the resolution. No motion sickness, no batteries, no cables around your neck. Everybody would switch and VR headsets would become niche product.
I don’t think that’s true. There are developments that really do constitute genuine progress: for example, going from CRTs to LCDs. It’s hard to imagine a world populated by high-resolution, energy-efficient, colour-accurate flatscreens suddenly being replaced by the invention of a gigantic ugly box that performs worse on every available metric.
…and yet CRTs still exist as a ‘niche’ product. There’s a whole corner of the internet where people admire and collect them.
It may well be that ‘spatial computing’ or whatever you wish to call it is going to be a genuine step forward, but it’s also easy to imagine it might not be. And that’s not true for every technology.
Except motion. In which twenty year old monitors on my desk make everything put next to it look abysmal by comparison.
I think the most damning "thought experiment" is "do you think you'd buy this?". I'm waiting for a VR headset for this. Less 'in office' and more 'WFH' but still same idea. I don't know if the Apple headset is the one (because my work/Apple won't let me connect my existing computer to it) but it looks to closely solve the problem on a hardware level.
I've screamed it from the mountains. I want a VR headset that acts as a monitor/tv and lets me turn any flat surface into a private desk. I travel a lot, and it sucks not having a monitor in a hotel. It sucks needing the extra room for a desk when I have tables already. Housing in my city is $1000+/sqft, so space for a desk or office is a meaningful expense for many. It sucks that having others in my house (kids, spouse) means I can't have a private, distraction free area without paying for a whole extra room (again, a premium).
Plenty of people besides me live in an n+1 bedroom house, where they have a whole extra room for a desk. Plenty more people live in urban environments where they can't afford extra private space. Think on the global scale, where people have more multi-generational homes, and live in smaller urban apartments. Sure they might not be able to afford the top-line apple headset today, but think longterm. Plenty more people are students, in academic halls and dorms and libraries, and would benefit from "on demand" distraction free, large monitors but don't have the private space for it.
I have a 48" monitor at home. Its basically a TV on my desk. I absolutely adore having so much space while I'm programming. If the resolution, price, battery life, weight and sweatiness issues of virtual headsets were all addressed to the point that I could use a headset like that instead of a giant monitor, I'd totally buy one. Especially if it meant I didn't have to lug my laptop around as a result. After all - the apple headset has an M2 chip in it. There's no good technical reason to need a laptop as well.
The apple vision headset looks like its getting much closer to that dream. The displays and the software stack look much more appealing than the quest 2 I have. But they need to address the weight and battery life. And it needs to be a lot cheaper (or a lot more capable) to justify its price. I'll probably grab one in a few generations if they keep iterating on it.
I complained to customer support about it - and they did nothing. Eventually I wrote a little background process which makes the whole display flash white every 5 minutes to stop it from dimming. Its a little weird and distracting - especially when I'm in the next room and don't expect a flash. But it works great.
The display might get burn in in a few years as a result. If it does I'll look for a replacement from anyone but gigabyte.
I turned this off in the hidden factory settings for the LG OLED: https://old.reddit.com/r/OLED_Gaming/comments/sbords/how_to_...
Perhaps there is something similar for yours too.
If you have a device that looks and feels like traditional glasses, but enables you to create and place an unlimited amount of virtual displays of any size and shape you desire, which can follow you around as you walk, why would you ever want to replace those with physical displays?
Certainly, I would have some uses for such AR glasses, but I wouldn’t like to wear them all day unless they are literally (and completely unrealistically) invisible and weightless. They won’t replace displays.
What I’d really like is paper-like non-emissive, high-contrast, full-color-gamut, millimeter-thin, wireless multitouch displays I can hang everywhere. Those are similarly unrealistic though.
VR headsets are just a stopgap until tech or price/power lowers enough to make AR more than a gimmick.
AR is the future, and everyone will have to have it the millisecond they start watching a demo of the Super Bowl being played out in your living room. Or pornography.
I quickly narrowed down on VR experiences that provide a decent feeling of facsimile to real life being the most interesting and engaging. Beat Saber is a case in point here but for me it was combat flight sims using stick, throttle and pedals. Moving to gestures and eye tracking wrecks all of that.
I also don’t know why the original article calls using a mouse indirect as if we aren’t masters at that as well. As well as a spatially aware species we are also a tooling using species. We are experts at leveraging our spatial reasoning to manipulate tools in the world to do things we otherwise couldn’t. A mouse movement in the world translating to moving a cursor on screen is a perfect fit.
I suspect that we are going to see most apps centre around this finger/thumb motion.
You have those when moving a mouse while holding a button, but that's not related to UI, you could just as well use it to do nothing, feedback is the same - NOT from the UI
Otherwise your fingers have the same things, so any gesture involving those is feedback just like the one you get if you touch non-finger object like a mouse
All of this is exactly the same - you have no tactile UI feedback
We can't even arrange icons on our iOS home screens as we please — we have to stick to a predefined grid with predefined sizes and use a terrible interface to move them around.
I do hope this will change in the future: our brains are wired for spatial memory and we should be making things easier for us by building software that is subservient to our needs, not the other way around.
For doing proper work the keyboard/trackpad combination looks to be ideal.
I just cannot get it to reliably recognize pinching. It's like only half the time, and sometimes I have to pinch like ten times in a row until success. Or the act of pinching also moves the hand off the item you're trying to click.
And then certain things your on-screen hands just can't reach. Super unclear why. But I think something to do with trying to map hands close to you, to an interface button visually one or two meters away.
The Vision Pro allows you to manipulate things that are too far away to reach. I don't know if this will "feel" good or not, I won't know until I get one in two days, but it will either feel like a superpower - or be annoying.
Or possibly first one, then the other.
I've alway been on the fence about VR, even for games, and never found any experiences to be compelling after a few minutes of fun.
However, one thing that struck me about Half-Life Alyx was the "magic" feeling of the Gravity Gloves in the game. In the game you point and reach for something, flick back, and it flies into your hand. It's very satisfying and intuitive, and possibly my favorite aspect of the design of that game.
I'm not sure if this would translate well out of a game design and into "real world" mixed reality, but it's an interesting thought.
So hey why not in VR/AR too ? Then I can easily levitate an entire pile to check thru it, like the way I can click on a stack on my Mac.
When I first started talking and writing about it, I described what we did as spatial computing because - well - its computing stuff in a spatial sense. I even went further, suggesting that the patterns that emerge from entities operating in a physical space resemble the mechanics of other "thinking systems" like Neural Nets. (see https://hivekit.io/blog/building-ai-without-a-neural-network...)
I felt, the term spatial computing was super applicable for that - and I still feel that way. But with the release of the vision pro, we had to surrender the term to Apple and go with the less pompous "workforce orchestration" instead.
Although I think part of the bleh feeling is just having to subconsciously adapt to the imperfections of a headset. For example, you can't just glance at anything and expect to see it, you have to shift your head for some things. Or like the headset itself moving ever so slightly on the face and your eye moving out of the way of the center of the lens. It gets exhausting and you don't know you're doing it unless you're actively paying attention.
I will say most of the advertising shows people interacting with 2D interfaces in 3D space so maybe this is how they're getting around it combined with just better build quality, better thought out straps, etc.
One thing I noticed was that My brain had two visual maps competing against each other. Reality and the virtual world. I sometimes found myself feeling the sensations from the virtual world from time to time while I was not using the head sets. It went away with time, but it was interesting anyway.
Evolution: opposable thumbs => front panel loader => command line => GUI => (you are here) => spatial => true cyberpunk cyberspace.
Tired: "Two dimensions ought to be enough for anybody."
1. R1 chip, which means that the spatial tracking basically never fails and is way more stable. Typical VR headsets will start lagging with the more CPU load they have, which causes disorientation and dizziness.
2. Eye tracking and gesture controls. There are cameras point at your eyes, down, and far to the sides. This means that no matter how you're sitting or where your arms are, it can see you "click" with your fingers. This level of seamlessness is necessary for the headset to feel effectively invisible. Typical VR headsets require you to deal with external controllers, which makes them more like consoles. Makes it more awkward to do simple things like check your phone or grab a drink.
I honestly don't think it's going to take off because nobody has demonstrated a "killer app", but the AVP is substantially different from other headsets in the space.
Are you confusing dropped frames with spatial tracking loss? Apple hasn't invented a technology to never drop frames so I expect once we hop into fully immersive Unity apps we'll have familiar stuttery moments.
That said, I know Unity's Polyspatial does have a flag to update the draw for each eye individually, so it seems like some disorientation should be possible, but it would be coming from inconsistent data across the eyes, not the way the system processes your head's orientation or the passthrough.
I can see how you could write a bad full-screen app though. Many ways to disorient in addition to inefficient draws. The app store approval process is only going to help so much there.
Full apps don't appear to use the Polyspatial system but its hard to tell exactly.
- It can leverage the Apple app store. While android headsets can sideload and run Android apps fine, none have the support of the Play Store.
- No controllers is a bold choice. Didn't work out for the hololens but should be better tech this time around.
* 6DOF tracking quality
* Ability to integrate with a Mac
* General high quality Apple software
* The hand and eye tracking features
I still don't think it will catch on. From all accounts it is less comfortable to wear than other headsets. Even with lighter headsets like the Quest there's literally no way 99% of people would choose wearing that for their normal work day over just using a normal computer. It's nowhere near comfortable enough.
People tolerate it for VR games because that's the only way you can play VR games. Apple seems to be targeting the AVP at stuff that you could just do on a laptop already.
One, with software, you could delay any action for a set amount of time as such to give your brain time to make a more concrete decision. Idk how long that delay needs to be, but I'm guessing it's less than 300ms.
On the other hand, if the hardware is responsive enough, you can simply decide to close the program/ad/whatever another split second after that fleeting thought.
The pessimist in me thinks that the first option is probably the best one given software optimization in the past two decades, but who knows.
Is there a type of software that already does this? Because it is not like it could not be implemented right now, nor that it would not be useful right now for the exact same reasons. Of course it is completely against most corporations' interests, so not sure who would produce that in the first place.
Several email clients including Gmail have the ability to have a short delay from clicking “Send” till it actually does the send to allow you the ability to undo.