VisionOS developer docs and Vision Pro SDK
developer.apple.com
developer.apple.com
It's upon reading these words that I realized that every computer I've owned was missing the ability to flip the bird to application windows as a means of force quitting. I'm not ashamed to admit that I'm anticipating that feature.
Consider how environmental factors impact your app. The characteristics of your physical environment can affect system load and thermals of the device. Consider the effect that ambient room temperature, the presence of other people, and the number and type of real-world objects can have on the your app‘s algorithms.
If you could track hand movement 1:1 enough that interacting with a 3D space was as natural as it would be as if it was actually in front of you, the problem is definitely a solved one at that point. If Apple manages to do this at any point in the lifecycle of this product line, it'll be a significant breakthrough for this type of computing / experience
Further, you can absolutely make 3D objects able to be grabbed, thrown, resized, etc. and it is actually pretty simple to do.
This is a button which has no physical presence at all.
Maybe Apple deliberately decided against including optional controllers, to make it clear that they are aiming at AR, not VR gaming. But even in AR the lack of tactile feedback could be jarring.
This all seems like playing Quake on a touchscreen.
A comparison could be drawn from the distinction between the Blackberry keyboard and the iPhone touch surface.
Apple’s design approach could well be aiming to make the Vision Pro feel as natural as possible.
From getting started in 2015 with the DK2, I've held from the very beginning that controllers are a dead end for VR. I've given countless demos where people are just completely confused because it's so unnatural and awkward, and it adds to the overall cognitive load to the point of destroying presence. Sure competent gamers can figure it out in a few minutes, but that's not the point. You don't interact with the world using triggers and buttons. You interact with the word by interacting with it. The controllers need to go away.
There’s ton of stuff you’ll never be able to do in VR without some kind of controller with buttons and joy sticks. Making the controllers smaller, lighter, more ergonomic and fairly standardized is the natural path forward. People who grow up with VR and games won’t have a problem.
The problem is the best apps attach all sorts of weird things to weird buttons.
I've given plenty of novices a try of a headset and it's a train wreck with them accidentally pushing buttons every which way that completely destroy whatever setup you've done. My favorite is devs like to make the "grip" buttons (on the side where your middle finger rests) shift the frame or scale of world (eg: OpenBrush does this). It's super natural once you know what you are doing, but give that to a novice and it's incredibly hard for them to learn to not grip the controllers. They cling on to them like crazy and the whole world is spinning around and they immediately tell you they are disoriented and nauseated and hate VR.
I'm happy the VisionPro uses gestures to control the interface, but I also agree with the parent poster that we'll need hardware controls someday. No matter how good the gesture recognizers get, waving your hands in space is never going to be as good as having physical controls and buttons attached to your hands.
As it stands, it's a dilemma. The Vision Pro is $3500 and still only does half of what I want a headset for.
The only saving grace is the Vision Pro is so expensive, I can pick up a Quest for only $500 and it will do the other half it's hardly any more ;-)
I've no doubt there will be tons of third, and possibly even first party peripherals for it, but treating gesture interaction as the baseline to design against was the right call. It's the number one thing that will help adoption right now if done right. I think the callout made in the keynote about each new Apple product being based around a particular interaction paradigm (e.g. mouse/keyboard, clickwheel, then multitouch, now gesture) makes it seem obvious that this is the natural evolution of HCI.
> No matter how good the gesture recognizers get, waving your hands in space is never going to be as good as having physical controls and buttons attached to your hands.
I thought this as well 5 years ago, but pose detection and reverse kinematics have come lightyears since then thanks to ML. I'm fully confident that what Apple is shipping with Vision Pro will be equal to or better than the 6DOF control fidelity of the gen 1 Oculus constellation based tracking. The only problem that remains is occlusion when reaching outside the FOV, which indeed it's hard to imagine a solution without controllers or an outside-in tracking setup.
For me, controllers are bad simply because your hands are full. You're giving up interacting with the real world to interact virtually. Having to figure out a remote control isn't all that insurmountable for the majority of the population but no one wants to hold a controller all day.
The missing haptics and discrete button feedback is an issue with hand tracking though.
That said, the sensors and software in this headset might finally be up to the task. And despite being unnatural it might be easy enough to pick up.
But I found it promising that the Vision Pro can see your hands in front of you as well, which should really help ease the learning curve, and prevent first-time users from feeling frustration with the UI.
Like when you grab a virtual object and it feels like you're grabbing absolutely nothing at all? This isn't natural and it's actually quite jarring in my experience.
Anyone have any ideas?
Had a similar issue after doing the Xcode 15 beta 2 download with both iOS 17 and visionOS 1 simulator checked. When creating a new visionOS App it would tell me that I'm missing the device/simulator support and prompted to install it. After confirming this it would show a download for a build that ended in xxxxxxxg instead of xxxxxxxf as was part of the download, so I assume that part was updated in the background and now is no longer the correct reference.
HTH
Now that this is definitely not a new idea, but I'd imagine, in the future, if the metaverse does sprang up, maybe we will see the same thing: Hey John Doe here is a widget which is a small window in the metaverse room 211, please write a script and make it interesting, you can do anything.
That definitely sounds very interesting. Man since I have a macbook pro at work, I might actually try it out, at least in the simulator.
One is mostly a list of examples of apps that have integrated; the other has a table of contents for the developer guide.
Have you tried reading past the first couple of paragraphs?
So, if I setup xcode, I can install the sdk and run the Simulator?
have you never heard of the Apple NDA before now? they are pretty much famous (infamous?) for their NDAs.
Didn't realize that extended to the creator of Flappy Bird!
If your goal is to hack around and try to get anything at all to work it might be fun, if your goal is to actually build a working VisionOS app, I'd probably suck it up and spend the $1000 on used MacBook.
The closed development ecosystem of Apple is alluring because it offers access to its iOS devices.
But what makes it worth the effort to learn a new SDK, or even buy a MacOs device to develop for it, for just a few thousands of devices?
It is the same type of a bet when deciding to start a tire company or a car mechanic shop shortly after the first cars ever were introduced on the roads. You don't bet on the exisitng base, you bet on that base expanding extremely wildly, and you want to secure your position early.
Also, I fully expect to be positively surprised by the sales of VisionPro. Keep in mind, we live in a world where one of the fairly popular flagship android phones (Galaxy Fold) sells retail for close to $2k. And it is still fundamentally a smartphone, it just folds and has a couple nice features due to it that others don't. But none of it is practically groundbreaking. I would be honestly even more surprised, if Apple wouldn't manage to bring a sub-$2k version of their AR device in the following few years.
A lot of people (and billions of VC) made that bet in 2016 with the first wave of headsets. It didn't pan out, and they got burned (myself included).
Here's hoping Apple is in this for the long haul, as they are basically the last hope for consumer VR reaching mainstream adoption.
It is clear that Meta tries to conquer the market from the low price end, which seems to work, as there hasn't been an expensive headset (like the Valve Index) which sold anywhere as many units as the Quest. Now Apple doubles down on the high price end route. It will be interesting to see which price approach proves successful in the long run, or whether they both converge at some sort of compromise. I wouldn't say Apple's approach has an obvious advantage.
This is kind of my point though. Over a hundred billion dollars invested by Facebook to sell 20 million units over 5 years (all at or below hardware cost). That's still a tiny drop in the bucket compared to mainstream adoption in the sense of iPhones and iPads. You could add up every headset ever sold by any company since Oculus 1 released in 2016, and it would still be less than the number of iPhones sold last year alone, both in units and revenue.
We are still waiting to see whether it will happen for VR. Apple just announced the first true second gen headset, so hopefully that does the trick.
It launched my career, but it’s such a poor comparison point for success. The iPhone replaced existing phones, it wasn’t a de novo market, or even technology really.
I didn’t review your FB #s carefully, but my guess is it performs some sort of coarse operation to guesstimate incremental employee cost due to VR
VR has absolutely "succeeded" in the sense that it's a neat toy to play games with that we didn't have before. But for it to be anything more than that, yes the iPhone analogy is apt. Otherwise it remains a niche PC gaming peripheral, and nothing more (i.e. where we've been stuck for the last 5 years).
Windows Mixed Reality headsets came out October 2018 with inside-out tracking. Oculus Quest came out May 2019.
The Vive technically could do camera passthrough, but I never saw anything actually supporting it well and it wasn't very high quality at all. The Vive Pro which also released before the Quest even had two cameras for stereoscopic passthrough.
For comparison to another gaming device, that's more than the Xbox Series X and Series S (which both came out around the same time) combined, and over half of the PS5 sales numbers. These are well-established products with loyal fanbases.
If you can, you should totally give it a shot.
If the Vision Pro replaces the phone in your pocket then maybe it's worth the price. Have fun putting it on in the coffee shop to take a call.
VR devices are also not optimized for general computing (as a sign of it, the Vision Pro won't natively support mac apps either)
Sadly I bet Apple won’t allow it this time around.
*I’m thinking of a comic that picturing Donald Duck’s uncle swimming in a pool of gold coins
Specifically 8 copies -- which I absolutely agree counts as "many copies" for what it was!
Is the Vision Pro actually that expensive at $3500?
A high/end LG OLED G3 77 inch TV @ $4,300
A Dell XPS laptop @ $4,000
A 2005 Honda Civic with 225,000 miles @ $3800
NBA season tickets mid-“club” level seats @ $4300 (does not include parking, dinner, transportation)
A Barcalounger power reclining loveseat @ $3120
An 11 x 18.5 ft outdoor storage shed made out of plastic and genuine steel frame @ $5274
You might need that Honda if you are barely scrapping by, but if you are like many HN readers then you know damn well that most of what you really needed was taken care of a long time ago.
Disposable income is disposable how you see fit.
I see fit to buy a Vision Pro after a short test-drive. And I expect to be amazed.
A high/end LG OLED G3 77 inch TV @ $4,300: Buy one of them to watch shows with your entire family. Want do do that with the Vision? You're either taking turns or forking out $14k for 4 of them.
A Dell XPS laptop @ $4,000: Can be used for many more things than the vision pro seems to be capable of. This is a work device, if you bought the vision pro could you _get rid of_ your work PC or would you need to have it anyway?
A 2005 Honda Civic with 225,000 miles @ $3800: I challenge you to trade in your car for a VR headset. Go on, tell us how much better your life is afterwards.
NBA season tickets mid-“club” level seats @ $4300 (does not include parking, dinner, transportation) A Barcalounger power reclining loveseat @ $3120
These two are actually comparable in that they're ultimately unnecessary luxury items.
An 11 x 18.5 ft outdoor storage shed made out of plastic and genuine steel frame @ $5274:
I'll just keep my lawnmower inside next to the shelf I keep my Vision Pro on.
These comparisons are ridiculous, I notice you didn't bother mentioning how cheap the Vision Pro is next to a house?
This is a new official platform from Apple, not some random Google experiment. They don't do something like that willy nilly. This will be supported for decades to come now, and be the basis for who knows how many products. It's about getting in at the ground floor, a la iOS circa 2007.
Not all Apple platforms are enormously successful for third parties.
This is different, this is a potential replacement for their core products.
AR/VR is far from that. I hope Apple's entry boosts it a lot further, but comparing it to the iPhone feels like there's only disappointment in waiting, even if it fairly succeeds like the Watch does for instance.
AppleTV was supposed to be a new thing with people developing entertainment apps and games that fit with a TV, not for general computing.
The watch added to the iPhone and had unique apps to take advantage of the sensors.
Neither of these were tied heavily to a macbook or other general purpose device. Unless you mean computing as in anything digital that runs code. Which I wouldn't really call a platform.
VisionOS feels closer to an AppleTV. New hardware with a different paradigm that needs a unique API and new apps built for it.
That said it does seem more like a supplemental device to your macbook + iphone. I doubt many are going to buy it with the intention of it being their main means of computing.
Maybe define what main platform of computing and core product mean if you could?
iPhone is a different class of market completely, not everyone needs a TV media center, everyone needs a phone.
I feel apple is better on this play to be more like the Mac or iPhone, than AppleTV or AppleWatch
We can never predict the future and can safely assume this will never have the reach of their phones, tablets, or computer market. But it is tempting to play the App Store lottery before the clones roll in!
Since it requires manufacturing scale (these things aren't going to be hand-built by a blacksmith in Germany) I would bet it isn't merely marketing smoke & mirrors.
Which is what my Vive does these days.
> Next month, Apple will open developer labs in Cupertino, London, Munich, Shanghai, Singapore, and Tokyo to provide developers with hands-on experience to test their apps on Apple Vision Pro hardware and get support from Apple engineers. Development teams will also be able to apply for developer kits to help them quickly build, iterate, and test right on Apple Vision Pro.
Wonder what those dev kits cost and how available they'll be. Apple probably doesn't anyone getting their hands on the early hardware/software using it for reviews before it's released.
Presumably you return it after release and get the money rolled over toward purchasing a production unit.
BTW, Android is not as bad because there are emulators, not just simulators (that is my understanding for the disparity).
To the best of my understanding, there are ways around it[1].
I believe you needed to be insane, and drunk to survive it.
On the flip side you do get a sense of the full size of the Universe.
The lack of a 60/120hz mode will cause inevitable stutter, but that's going to be the case with any 60hz content (24p will be fine due to the 96hz mode).
Or, I dunno, they could single-handedly port Half Life Alyx. But I feel like the virtual CRT thing is more manageable.
CRTs also had no native resolution. They would instead change resolution physically, which made fuzzy interpolation unnecessary.
For a long time, they also had much higher refresh rates and better contrast than LCDs, though this has been matched in recent times.
CRTs didn't have a native resolution, but colour ones did have either masks or grills with individual RGB subpixels just like a flat panel. There was still interpolation in a sense, it was just done physically instead of transforming a framebuffer. If you made a CRT photon gun accurate enough that it could reliably address those subpixels it would then have a "native" resolution.
e: Also, a heads up that VR headsets strobe by default, it's part of what makes them work at all. Even LCD models, where they strobe the backlight.
It's already quite effective in EmuVR even on lower end HMDs, get close enough to the virtual TVs and you can see the individual RGB phosphor dots. Moire is pretty bad at a distance though.
This, but with movement. Also do pose detection on others or a mirror and see the organs move in a scientifically accurate way.
---------------------
Bring real-world objects into VR but render them in the style of the scene which already exists. (99% sure not possible with this API). Basically, if I'm playing a flight simulator/etc, I want to be able to see my water bottle, keyboard, mouse, cell phone, etc so that I can easily grab them. But I don't want the immersion to be broken.
---------------------
Track conversations I have with other people and use an LLM to remind me about the relevant conversation history we've had.
----------------------
Years back I did a "startup training" weekend. My team decided to use my idea of gamifying museums and historical sights. The team was fun and had some good puns!
I'd originally thought it'd be fun to play a version of the "floor is lava game" at the ruined temple where they partly filmed the Temple of Doom. I still wish I could do that someday! A sort of historical scavenger game would be really fun way to engage the history of the place.
So custom image texture on a model with some animation, so you can checkout how a flag design would look when flying.
Another similar idea with object recognition would be an AR scavenger hunt.
> Dwell Control will provide on-screen options for Tap, Scroll, Long Press, and Drag, allowing for users to interact with the interface without using their hands.
Does the article address any of your concerns?