Why Surgeons Are Wearing the Apple Vision Pro in Operating Rooms
time.com
time.com
Everything from:
- overlaying explanations, diagrams, realtime simulated current flows onto circuit boards as you are looking at them
- appliance / home repair
- juggling/sleight of hand - its significantly easier to learn these kinds of skills from a human (or in this case a virtual human) so that your angle is the same instead of a video where its reversed
- artistry - place a model of something in the center of your living room, and sketch on a canvas from it as you walk around it
And these are just a few things I came up with based on my own personal hobbies. Everyone who worships at the proverbial alter of Apollo will likely have their own wishlist.
I'm highly, highly skeptical. There is a lot of evidence that people (especially children) learn much better through low tech approaches than higher tech ones (I'm pretty sure there was an article on HN recently comparing how kids learned when reading paper books compared to screens).
Old school pen and paper and physical books tap into tactile learning. Even writing on my tablet isn't as good as just scribbling on some scratch so I'm definitely team analogue.
But its hard to not get excited for the prospect of being able to bring stuff like the dinosaurs and space to life for kids.
Even something like obd2 readers is tech honestly. The other day I took my 8 year old out to my car as it was not running right. I showed him how to use the reader. Then showed that my car had a p0304 code. I told him that is a misfire cylinder 4. Then showed him how to open hood and where my coils are and how the car had 4 cylinders. Then explained why cars misfire and that often it can just be the coils so test those first. Explained how we could swap coil 3 and 4 and check the code again. We did that and when I plugged reader back in the code was now p0303. My kid immediately yelled the coil is bad! Because he knew that since the code moved when we moved the coil that is the problem. All that said I do think yes tech can absolutely be used to improve learning in some situations. The obd2 reader is a bit basic yes but if we had AR I can think of dozens or hundreds of applications we could make from it. If I could afford it I would use it with my kids.
I don’t think AR is going to teach kids to read, but augmenting instruction in things like the trades, sciences, and sports all seem likely to have strong benefits. Experiential educational opportunities like museums and historical sites also seem like likely targets.
A good analogue is video. Video can be a great tool in certain subjects- I still swear everything I know about protein synthesis is from a video in college- but it is generally used as an secondary rather than a primary tool.
The video is from the 1970s and protein synthesis is explained through interpretative dance and and poetry. Its truly amazing https://youtu.be/u9dhO0iCLww?si=lTO95vbysVTB6DLa
One teacher put the extra effort to actually use the equipment. The plan failed miserably. The core of the issue was that getting a bunch of teenagers to learn things they don't want to learn requires divine intervention, not a single underqualified teacher on a mission because all the qualified ones have better career options than working themselves to death for what amounts to one and a half minimum wages in a trenchcoat but no government will put serious investment into education because such projects span multiple elections.
You don’t need augmented reality for that, Haynes manuals still exist, or you can just google it, there’s also stuff like Alldata…
Motorsport simulations and golf come to mind
Not just for leisure but a serious aspect of professional practice
Visually reinforcing the difference between a 0.1 second window for motorsport, or thumb sized threshold for a golf chip or putt, for example.
I once had to learn to juggle for a stage play. I could have sat though a hundred fancy presentations on how to do it, and fail abysmally as it is all about feeling and timing, things that are even sabotaged by AR latency.
Artistry? Now there's the real AR market. Not the Metropolitan paint stroker kind, but the one that usually ends in Hub.
I can only speak anecdotally, but there have been dozens of times where I've had to fix something and I had to constantly go back-and-forth to a corresponding tech manual PDF scrolling up and down through hundreds of pages, makes AR very worthwhile to me.
For juggling I was referring to people who essentially can already juggle and want to learn new patterns. It's always going to be more difficult if you lack the requisite fundamental basic hand eye coordination and timing to be able to juggle at all.
The latency aspect is interesting, but if you've ever played or had any experience with video games, particularly rhythm games, such as DDR, you'll know that the human brain is highly adaptable and able to recognize and compensate for latency. Additionally, latency isn't really a thing in juggling where it's not necessarily about the split second reflexes, it's about pattern recognition, your hands are usually already in the approximate right place for the ball to fall into the hands.
I also think there's a great space for progressive desensitization to help treat phobias with AR since you can effectively conjure anything at any level of sensitivity.
I guess, as is almost always the case - those with inherent curiosity will be able to take full advantage of these types of tools. But I'd like to think that it might inspire people to try new things.
And honestly if pornography drives mass adoption which results in corresponding advancements and lowering the price, then I'm OK with that deal.
On the other hand, enhancing reality for enhanced awareness and data is something I think is very promising. We already do it all the time, putting light indicators or feeling the heat are things people are used to do to monitor and understand systems. Imagine how useful would be to integrate proper sensors and device connections? They are doing it for fighter pilots but why don't do it when we deal with any kind of machinery?
Never mind being hacked, what data is being sent to and collected by Apple? Why should I trust them? I'd be furious if I found out my doctor was using a product like this without my knowledge and consent. Even if you, for whatever reason, actually believe that Apple and every Apple employee or third party contractor who might ever have access to your data is 100% infallibly honest and incapable of error once VR headsets become common in the surgery other companies pushing that kind of tech will want in on the action including facebook and microsoft.
Have they shipped telepresence where the camera feed of one headset is live streamed to another? For getting expert advice remotely this does sound like a killer app (or should I say lifesaving?) , like having a voice in your head who is situated in the same viewpoint as you
But once you do you can access the camera feed, compress and stream across the world.
I’m guessing that /s applies here; but since I’m married to a surgeon and through her, understand that the truth around that, and around advice-taking and -giving behaviour in and out of the OR is far more nuanced than you might imagine.
I think iterations of Vision Pro will take years and the final cost will not be what it is today. What everyone sees today is a developer toolkit that also serves as a public demo. It is not a consumer product that can be scaled (yet).
First, when I got my first iPhone, I was amazed by it. I used it all the time. Yes, there were missing features and shortcomings, but many/most of those missing features were obvious, and it was actually pretty delightful that with every new software update I'd cross some of those missing features off the list.
None of that dynamic exists with the Apple Vision Pro. Even big Apple fanboys end up leaving it unused after the novelty wears off because it's fundamentally a solution looking for a problem. People aren't leaving it untouched in a drawer because there are just a couple missing features they want.
Now, all that said, as much as I would dread having this computer strapped to my face as a "normal" person, I think the use case given in this article makes perfect sense. The surgeon is doing a specific task, and the Vision Pro helps with that specific task. While I believe the Vision Pro will fail as a general consumer product, I hope it can live on for these niche use cases, although I don't know if the economics will make sense for such a relatively small market.
If you gave me an Apple Vision Pro, I literally have no idea what I would do with it. I would really love to get into something new like this but I literally can't think of what I would do with it. It is a technology that seems unreasonably useless. It seems like there should be many more use cases than there is.
TBH the dynamic you're describing would have been Meta's glasses if they also release a viable version of Orion as the second product. But at this point nobody's in a place to make it a reality, the fact that Apple actually released the Vision Pro is the most telling sign that they don't see themselves reaching that level anytime soon.
Meta hasn't sold tons but 700k for a device that needs to get past people's privacy concerns is decent and then Meta showing off Orion .. where Ray Bans are headed in a few years i think is pretty cool.
The elephant in the room is that we're already really good at human-computer interfaces. VR not only isn't an upgrade, it's a downgrade.
The precision of tapping in 3D space is much lower than on a screen or a keyboard. Typing in VR becomes 10x harder. Voice isn't a solution either, because talking is always slower than typing. And for gestures, swiping on a phone is already really fast. Swiping in 3D isn't any faster.
The most precise and fastest is, of course, keyboard and mouse. Touchscreens are a step-down, but one we took for the advantage of super tiny devices you always have on you.
VR is certainly not tiny. And, even if it can be and you can just wear it whenever - what's the benefit? Going from a laptop to a phone is obvious, but a phone is already very small and already always on you. What do you gain then from a VR device? I think probably next to nothing.
Everyone important left years ago for Meta and Apple, or startups like Magic leap. There have been several exoduses, whether it was due to bad leadership (see below), military contracts or just a program that was never treated more than an R&D outlet like the Kinect DX was too.
Then the remaining head of program was fired for watching porn during team reviews in headset. Among other allegations of impropriety. Before that, he was considered an awful boss to work for by many.
With nobody left, Microsoft had a ghost project that was eventually killed off. Anyone in this space knew it was coming just by talking to any one on the team.
Its death means nothing more than Microsoft has once again shown it can’t stick with things outside its core competencies.
The HoloLens was also a significantly more limited device because of the nature of waveguides. There are definitely advantages to additive displays but if anyone’s used a HoloLens 1/2, it’s a far inferior experience to the Vision Pro or even other waveguide based products like Magic Leap.
Chicken or the egg?
I don't believe that they couldn't find more people to work for it. Pay double, no one would leave. If they let a whole project die, they just weren't interested enough.
For some of these roles, it’s highly specialized people and you can’t find just any engineer to replace them. When your vision leaders leave, you can’t just find someone to replace them. When they’re leaving because the head of program is an abusive pervert, money isn’t enough to keep them.
And this is a recurring theme with anything outside of Microsoft’s core competencies. They cannot long term execute.
Windows and servers has been their competency since the 90s and has been firmly stuck there.
Mobile computing? Failed twice.
Music players? Failed.
Xbox? Currently in the process of failing for the second generation in a row.
Microsoft do not have the necessary culture to do this. They leave things on the vine and see if it dies or blooms. They don’t weave a story or make strong ecosystem plays.
That’s the biggest difference between them and Apple. An Apple product comes with a vision story. It comes with the ecosystem.
She was later told that this is a pretty typical experience if you end up in a lower priority team that’s primarily made up on contractors.
They went from dominating in the 360 era (when Sony fumbled the PS3, and Nintendo switched directions to the Wii) to two generations of poorly received releases and being in third place. That’s even if you filter by the specific markets they dominated in with the 360.
And what are they focused on now? Windows gaming and cloud services. Their two core competencies.
And why? The people behind the original Xbox and 360 vision left. When you have no vision leaders in the company you fall back to what you know. In their case: windows and cloud services.
The gamepass and cross play is nice, but they don’t have compelling exclusives to drive hardware sales. Combine that with absolutely terrible product strategy like their naming strategy and rewriting feature parity between the S and X, it’s no wonder they’re third place now.
Gamepass has only further eroded their console market because it reduces the need to buy it when you can play on your computer or stream.
They’ve also barely capitalized on many of their extravagant purchases yet. Especially with Activision Blizzard, it remains to be seen how they’ll do exclusives without falling afoul of antitrust.
"Augmented for surgical success—a reality now"
https://www.accenture.com/in-en/case-studies/technology/micr...
"Microsoft HoloLens 2 in Medical and Healthcare Context: State of the Art and Future Prospects"
https://pmc.ncbi.nlm.nih.gov/articles/PMC9611914/
"AR in medicine: Spinal surgery via Hololens 2 shows the future of medicine"
https://mixed-news.com/en/ar-surgery-hololens-future-of-medi... "Providence Swedish advances brain surgery and presurgical planning with Microsoft HoloLens 2"
"Providence Swedish advances brain surgery and presurgical planning with Microsoft HoloLens 2"
https://blog.swedish.org/swedish-news/providence-swedish-is-...
"World Premiere Shoulder Surgery with Microsoft HoloLens"
https://www.evolutis-group.com/en/world-premiere-shoulder-su...
They’re just saying it does what they need better. A significantly higher resolution, image clarity and field of view than what the HoloLens could provide.
From the article:
> In previous years, Horgan tried other headsets, like Google Glass and Microsoft HoloLens, and found they weren’t high-resolution enough.
There’s another mention of HoloLens as well in the article.
IMO, Apple's most successful approach is perfecting an already marketed product and turn it into a global phenomenon. As this article pointed out, the AVP (not aliens vs predator) has some advantages and definitely an improvement. But this first version is still not improved enough to reach mass appeal. Perhaps the next one?
The tech will keep shrinking and the cost will come down. We are still in the early adopter phase.
But everyone that I know who has came away fully understanding what Apple is trying to do. Virtual experiences e.g. being front row at a sports game or concert, watching movies with friends in a theatre, climbing Mount Everest etc. And working with multiple screens.
If it can just do those two things it will be another iPad sized business which is still worth tens of billions to Apple.
That may sound pedantic, but actually it's a pretty huge difference. They don't want to sell a device primarily for closed-off experiences; they want to sell an iPad on your face that you can also, if you want, use for movies and whatever.
Why would anyone think your average person wants an iPad on your face?
Imagine a typical use case for a computer. Writing an essay, coding, doing CAD work, using Excel, replying to emails, ordering products.
Here's how much I'd want to do those per device:
1. TV - fuck no
2. Phone - prefer not to, but replying to an email isn't the worse thing ever
3. iPad - pretty good, sure
3. Desktop PC - by far fastest, most dense, and most convenient experience
4. VR - If it even can be done, I know it will be infuriating.
I give VR essentially zero chance outside of a few niches, like adult entertainment, virtual tours and the like.
The harsh divide between the world the body inhabits and what is presented comes with endless problems.
It is cool though. No joke.
AR, has a chance. It does not impose that hard divide and it can augment the world without adding latency to it.
Another problem you have is you are constantly asked to travel places to be physically present with people because apparently it makes you collaborate with them better. But because there's no other choice you just do it... it's not a problem to be solved, it's just a fact of life.
Obviously I'm making assumptions here, but these are the types of "problems" VR solves. By and large people don't acknowledge them because it takes too much imagination to think it could ever be different. I have full confidence that you'll dismiss all these problems right now in your reply - as much confidence as I have that when the tech reaches the right point that it truly solves them in a convenient and comfortable way, you'll embrace it and declare they were obvious all along.
Caveman like?
That seems way over the top.
Essentially making the computing experience part of us is super compelling. I get that.
But it all remains in very early, highly speculative days.
As for valuable space, whose systems do you really trust? The ones running in my home are known and not a worry, or untrusted and managed appropriately.
For it to work like you hint at, we need either:
Massive, central systems that people essentially rent and are forced to trust
, or
Small ones so power efficient they can be implanted and perform in meaningful ways using minimal hardware, ideally communicating along internal channels somehow.
Even with those, sharing is a big deal, potentially exposing a now very dangerous and highly personal resource to others.
Meanwhile, the cavemen sent people to the moon on far less overall computing than we often carry in our pockets these days.
Just for the record, I'm not even a little bit excited by this, nor do I expect I'll ever buy such a device. I'd want the full matrix experience before I went in, so that I could do things I can't do in real life. However, chances are I won't want to do those things any more if/when it ever becomes a possibility.
I also share your concerns about how this stuff will likely be implemented (ie. on their cloud). But tbh that's already lost at this point.
But I can't deny that getting away from keyboards, monitors, desks etc is hugely valuable.
I don't like the approach. I am not at all sure "should" is the right word to use here. It could be very compelling to operate that way.
I think it might be. And I want to try it, same as I have everything else to see what the kryptonite is and how it will matter.
The better, higher value way to move past keyboard, mouse and display is to talk up the new paradigm. People will use it because it is new. They will also use it because it nails some use case or other that is super important to them.
Leading with our existing interfaces suck sets all the wrong expectations and could actually impact the incoming tech!
There is no need to punch down on these things.
Setting the security and privacy issues, which are significant, we are left with value potential and what it will take to actualize it.
VR mostly sucks unless one is seeking immersive experiences. And there it shines bright indeed. Outside of that, the tech is jarring, high latency, low-fidelity mess when passing the real world through as is often needed.
AR eliminates that, while still being both immersive and able to augment what one is doing in various apparently considerable use value laden ways.
I think about CAD, as one example I am very familiar with. I have used pretty much everything ever made for CAD too. Buttons and dials, macro pads, light pens, foot controller, space controllers, keyboard, mouse, old school tablet, new school touch and stylus visual tablet, voice input,, and I could go on.
My favorite by far is a big, fast display 3D active shutter glasses capable. Modeling becomes more fluid and the user can actually see complex surfaces in an intuitive way. Assembly is fantastic when one can fly around the thing with a space controller while still being able to pick, trigger macros and just build.
Many would call a setup like that advanced. Truth is it is a nice Samsung 3D plasma in my living room and a laptop with a 3D controller and mouse plugged in, often on my couch for comfort.
Despite how compelling that is, few people do it.
Now, take away keyboard. How do we input specifics? Voice input? Waggle flanges at virtual keyboards and other devices?
We could develop more precise language, and borrow from the sciences and describe things letting the system create them.
Maybe. But there remains a ton of details to contend with.
Let's say I remain skeptical.
But does any of this make keyboard and mouse out dated, caveman like, or just bad?
Nope.
The way I have always done it is to just use the new stuff and see what comes. Weave that into my process and gain value without also increasing costs and risks.
Heck, I have even used a system with a full haptic interface! The software was called "Freeform" the company was sensable or something close to that.
Basically, it was all about clay and one could carve and paint on it using the haptic! One fun thing was to carve a hole, then set the tool into it, get up for a drink, with the haptic floating jn mid air just as if it were placed into a real block of clay!
I loved that thing and when I put little kids on it, they made surprising stuff.
But it was no real answer for much outside it's killer value proposition.
Few things are.
Maybe I will put this another way:
Every UX device has both a super power and one or more kryptonites to deal with.
Which explains why VR is not going to boom. And it explains how AR might too.
And should that happen, it won't be how shitty one may feel keyboard mouse is.
Nope.
It will be the superpower, and with AR that is the ability to overlay information onto our already keen senses
I know these were just examples, but my point is the problems you’re presenting are really contrived compared to stuff like being able to listen to a lot of music on the go, having a phone with an internet browser in your pocket, or a well-made digital watch that can monitor your health without efforts on your end. All those products had an extremely clear, immediate value proposition, and didn’t require a new viewpoint on my understanding of technology.
The Vision Pro, especially after reading comments like yours, kind of feels like a product designed by people that should have lifted their VR glasses a bit more often and just have taken a walk in the forest instead.
What about other places, like hotels? What about trains and planes?
Even at home, I wish I could get rid of my TV and make it disappear when I don't use it. These new display glasses already take way less space to store than monitors. If they get the tech to the point that it's as comfortable to watch as a TV and just barely more expensive, I could see myself getting rid of my TV and saving the space, making my living-room nicer. I'd also get a screen size I'd never consider for my living-room because of how much space it takes.
And that's for entertainment only, for which I'm usually reluctant to spend much. If these devices become good enough to do long work sessions on them, I could have exactly the same setup everywhere which would be huge for me. I could also get rid of my monitors and reclaim my desk space. I think that's when I'd buy such a device.
See also the comments from 3 days ago from people already doing just that: https://news.ycombinator.com/item?id=41859012
So in a way it's already here. Maybe not everyone will have one, so it won't be like mobile phones. I've never had a tablet either, that doesn't mean they haven't been successful enough.
You can make an argument about the size of the screen but remember! VR is, at least, a few orders of magnitude worse to use than mouse/keyboard. Having a big screen wherever is cool - now use it. Do you want to use a floating touch keyboard or voice commands to use Excel? Probably not, so best to just carry around a laptop.
For number two, we don't actually "know" what makes in person collaboration good (if at all). And we all disagree. But if there's a human aspect to it, then VR falls flat once again. Why not just use Zoom, and then we got a ton of other tools? If we're not gonna bother to be face to face, then okay, let's lean into better technologies that already exist.
Show me something on my eye glasses, let me charge them overnight, charge me $700 and you bet I'll be using them daily.
Just look at the unlikely success of the Apple Watch. Nobody really needs it but it's practical, it replaces a piece of wear that people already used, and it does not cost a fortune.
Tech just is not there yet for glasses.
Nobody can say for sure how long the Vision Pro will exist, or whether it will be eventually turned into something completely different. Saying with any certainty that it is useless is just as short sighted as predicting a runaway success.
personally it is hard to tell what made the surgeons found groundbreaking about this device, as the ability to see another video feed would have been equally as easy on other platforms, if not easier.