Show HN: I built an interactive cloth solver for Apple Vision Pro

Really neat, are you running the cloth simulation on the CPU or GPU? How many elements are in the simulated cloth? Good luck on future AR projects!

The cloth sim is GPU. It’s not that heavy currently, around 40k quads. You could go much heavier, easily 500k, but it also changed the cloth behaviour.

The limiting factor is actually not the sim, but the initial load time for the procedural geometry. I might try adding some subdivisions though..

Is the choppy movement of the cloth because of the limitations of the device or something else?

Can’t believe this is what $4-5K piece of tech looks like. Wild.

It's because of the simultaneous recording / screen capture - it's smoother on device with less lag.

Is the video captured using the dev kit?

Not to disparage OP or this product - which sure looks cool - but man, the idea of buying a $3500 device to simulate something I can do by opening my wardrobe seems…absurd.

Did you watch the video? I don't think anyone has anti-gravity cloth that lights up on touch in their wardrobe.

You’ll be delighted to know that the $3500 device can actually run many other applications!

Can it though? What apps are people actually using?

Apple is very much not having an iPhone moment with Vision Pro: there is no explosion of creativity or a new market rush. And there may never be one.

Apple treated devs like trash for years and now they have a new platform nobody wants to build on. Oops.

This is really cool! I save AVP for work but have been eager to find new meditation interfaces because it feels like AR has so much potential for this. Will try it out!

Congrats! What has it been like developing on the AVP?

Thanks!

It's been fun. It's not too dissimilar to iOS - a lot of spatial capabilities are linked very closely with RealityKit, it's worth looking at the API if you're interested. I was thinking we'd use Metal for rendering, but as I think there are privacy issues with accessing the raw camera data, it's only supported for 'full' spaces, so not the mixed / camera feed + overlay.

API's are a lot more limited than IOS however from my experience.

It seems weird that the interface seems to encourage interacting with it from afar and has no indicators as to where your actions would affect it (like a highly on a pickupable node). Is there some sort of intuitive reason for that that’s not obvious on a video? Not a complaint just curious.

Good point - I should have probably shown the direct interaction too (touching the actual fabric in space) - there's a gif here: https://jmp.sh/s/lHqJm6NEvqMqkXMPyUpZ. It was a little but laggy when also screen recording on device.

In the video I am looking at where I want to interact and then using a pinch gesture and the sounds are mapped to different cells of the cloth. By either looking and tapping, or playing directly you can hopefully play your intended sound.

So it looks for gesture events and maps them to locations where your eyes are looking rather than where the gesture was done?

Is that how most vision pro interfaces work?

Yes, that's basically it. If the gesture is targeted to something, either by looking at it or touching it directly in space then it responds. Direct touching takes priority, so I think you could look away but still touch in space.

Are you able to have it interact with the environment?

Say you slide it over so it's hovering over the edge of your sofa and then you turn gravity on. It seems like you're just a few steps away from what!

A few really big steps.

haha, yes - it’s definitely possible, just may be bit tricky.

You can get detect planes and the mesh geo of the room.

I imagined it would be a tool for designing clothes. It would then output cutting patterns or stitching instructions. That would be the solving part. :)

I’ve been trying to figure out if Vision Pro suffers the same fate as other AR glasses that it can’t be used in the sun, Anyone knows?

I was curious if the micro led display could withstand more heat than the Quest headsets but I can't find anyone testing (accidentally or otherwise) on an actual device.

As far as tracking goes, the AVP has downward facing cameras and a lidar system that presumably works as well in the sun as their phone lidars.

You can use it in the sun as it is using video passthrough not optical AR like the hololens.

question is whether tracking works - the sun and its pesky infrared rays overpowers any structured light projections (appears as noise)

It's not AR glasses. It's VR with lots of cameras.

You're describing an approach to implementing AR.

Simple test: can you get a drink out of the fridge while wearing the headset?

If the answer is yes, it's AR.

The definitions of these terms are certainly not fixed or universally agreed-upon, but on the spectrum from VR to MR to AR, the Vision Pro is definitely not at the AR end of things, and is very much not the class of device that the term "AR glasses" refers to.

The problems that "AR glasses" have with operating outside in full daylight are inherent to a class of devices that does not include the Vision Pro. To the extent that the Vision Pro works better outdoors than whatever "AR glasses" OP had in mind, it's because the Vision Pro is a fundamentally different kind of device with wildly different tradeoffs and design goals.

So to summarize: unlike other AR glasses, it isn't unusable in sunlight, because it uses cameras to provide augmented reality, rather than something else.

Got it.

Why "other" AR glasses?

AVP is not glasses. It's a headset.

Headsets and glasses are two entirely different product categories.

Because a refusal to understand what someone means when it's obvious, doesn't (as the delusion would have it) make that person look intelligent or win them imaginary internet points. It's just annoying.

You are correcting his terminology using a categorization (AR, MR, VR, XR, etc) which is not standardized or even commonly used. So what's the point of correcting?

The simplest common-use definitions are:

* Augmented: combines synthetic and real-world imagery

* Virtual: fully-synthetic imagery

AVP can do both.

Doing both is called mixed reality. The only caveat is you see reality through a screen and not normal gasses. The terminology is very common in the XR industry. https://www.interaction-design.org/literature/article/beyond...

No. Please see the definitions. You’ll find links to them all under XR since XR covers them all: https://en.m.wikipedia.org/wiki/Extended_reality

AVP is, by all definitions, XR, VR, AR, and mixed reality.

AR with cameras and screens is an implementation detail, with the definition being realtime. You can easily play ping pong with an AVP on.

From the title I thought you made a optical recognition system for sorting out the laundry, finding sock pairs etc.

Very cool sim, but kind of disappointed still. Will build this someday.

I half-joked on (then) Twitter about 15 years ago that if someone made AR that would help me sort my clothes by laundry temperature I’d buy it in a heartbeat.

Then a friend told me I could just wash everything in cold water. Crisis averted.

Haha, I wish - that would be great!

You’ll have to track all your clothing with UWB tags!

Have you seen the Uniqulo checkout process?

Willing to trade a barely-used Quest Pro for an AVP. ;@D

Im willing to trade a barely used AVP for ££

guys im interested in developing for Apple Vision Pro, what are some gotchas, are you seeing success, what have you put on the store?

Main gotcha: nobody owns one.

how do you track breathing for Lungy? Is it using the microphone?

I've just made a ShowHN for Lungy here, it has some more details: https://news.ycombinator.com/item?id=40553986

I literally haven’t thought about the Vision Pro even a single time in months.

Wild for such a hyped-up technology product. It just shipped and vanished.

Their stance on developer access made it DOA on day 1.

All apps are flying blind when you're multitasking: apps can't even use iOS style marker tracking unless you run them in "immersive mode", which makes them the only app running. That combined with no camera access at all, extremely laggy hand tracking, and an inability to do room scale without a constant passthrough make it somehow less capable than a Quest 3.

I bought a Quest 3 after my Vision Pro and it's a legitimately better piece of hardware except for the displays, and passthrough (which is gimped in usage on AVP). Even the lenses are better on the Q3. Meta has a commanding lead in VR after all

On the other hand, the weakness of the onboard compute significantly restricts the Quests’ potential, as does their inability to take a DisplayPort input from a PC (the tether can only serve crappy compressed video).

I own a Quest 2 and AVP and while the Quest is alright for what it does and regularly gets used (for PC-tethered Beat Saber mainly), I’m on the lookout for a quality dedicated PCVR-oriented replacement that doesn’t break the bank. I don’t see myself buying another Quest in the future unless they add back DisplayPort input or the onboard compute both becomes more powerful and gains the capability to run Steam so I don’t have to buy games from Facebook to play untethered.

Uncompressed 4k@90 isn't exactly trivial. I think you'd need TB3 or USB4 for the bandwidth alone and Qualcomm doesn't provide that on the chipset. The device itself can decompress 8K video if it's stored in disk.

I'm fairly certain it's not Meta standing in the way here. My bet is Quest 4 will be able to achieve this but we'll see.

Yeah, Meta has a decent excuse in that they're basically using a smartphone chip for their headsets. Apple's using a laptop chip and another big chip, so not having any high-speed IO for the Vision Pro is more of a glaring omission.

As far as passthrough permissions, I don't think either platform lets you near the feed. Vision OS has their nonimmersive 3d widget mode which goes further and only gives you input when your widget is being interacted with. I assume Meta will go a similar route there.

I can forgive most of the Quest's OS limitations because it's functionally an entertainment device.

The AVP should have been a powerhouse like the Hololens, and the $3500 price tag didn't exactly scream anything other than enterprise for mass adoption.

I'm pretty surprised by the idea the Hololens was a powerhouse. Can you elaborate on that? My experience with it certainly didn't evoke 'powerhouse.'

I had a pair that I ended up returning [1]. I think about them often, multiple times a week when I’m using my laptop or tablet. I felt incredibly productive with the AVP when writing code away from my main computer. And they were just really cool/fun to use. I hope to get the second iteration if Apple continues to make them [2].

[1] I bought them because I had several app ideas for AVP that I wanted to explore, but right after they arrived I promptly signed a new client and couldn’t justify the extra time to learn Swift and AVP dev. I decided to return the AVP before the refund window expired.

[2] I realize the irony of hoping Apple continues to make the AVP when the act of returning mine may end up helping contribute to its downfall.

Ever recorded any metrics? I wonder how much of your "feeling" productive is real.

Or if you live outside the US, just vanished, since it has never shipped.

My wife and I saw a Segway in Golden Gate Park yesterday, and I made a similar comment about it.

Those are so fun to use at Burning Man! Just don't run out of battery in the middle of the playa or you'll be lugging the thing back for a mile through dust storms

I use mine almost every day, it's awesome for traveling. It might be the best "watching videos in bed" device ever made.

This frightens me. I have been working at reducing my time of watching videos in bed, and all I have to tempt me is a phone and a laptop. It's such an easy trap to fall into.

Spending time horizontal while not asleep is very unhealthy for your heart, and many other systems in your body.

> Spending time horizontal while not asleep is very unhealthy for your heart.

Never heard of this before, any references?

I will try to find some references, but the basic concept is that your heart will atrophy when it doesn't have to work against gravity, as in pumping blood from foot to head, while vertical.

This is why standing desks are a thing, why an astronaut will lose 30% of his heart-mass after one year on the ISS, and why patients after a long coma have a hard time doing cardio.

edit:

"Cardiac atrophy after bed rest and spaceflight"

https://www.researchgate.net/publication/11883764_Cardiac_at...

"Cardiac atrophy in women following bed rest"

https://pubmed.ncbi.nlm.nih.gov/17379748/

Seems like it should be easy enough to offset with exercise though? Like if you’re doing cardio a few days a week you’re not going to suffer too much by chilling in bed a couple evenings.

Yeah, this seems possible. I was thinking of someone more generally increasing their horizontal time without doing anything to make up for it. This is my own projection, as it is what I have been guilty of doing.

I fell into the trap ages ago, and spend as much time horizontally as possible without actively dying.

You’re absolutely right that it’s bad for you.

> Spending time horizontal while not asleep is very unhealthy for your heart.

Why is this?

To be clear, it's not that sleeping while horizontal is better than being awake while horizontal. We all need solid sleep, but being horizontal should otherwise be minimized as it is the ultimate sedentary lifestyle.[0] This causes heart atrophy.

[0] https://www.betterhealth.vic.gov.au/health/healthyliving/the...

see my other comment for other references: https://news.ycombinator.com/item?id=40549893

I think you’re incorrectly jumping to correlations/causations.

Your references are due to sedentary, not laying down. You can have a sedentary life standing in place most of the day. Likewise, the problem is being sedentary, not what position your sedentary in.

Look at the logic in my comment link, does this not make sense? This seems like basic physics and cardio to me.

The less time that your heart has to fight against pressure/gravity, the weaker that muscle will become. This is pure causation.

https://news.ycombinator.com/item?id=40549893

It’s a reasonable hypothesis, but you’d need to experiment to validate it. It’s easy to imagine how prolonged exposure to bed rest or microgravity may trigger heart muscle remodelling in a way that intermittent bouts of lying down would not.

I read the second paragraph as a euphemism for having sex. I was confused because consensual sex is generally considered a healthy activity.

The thing is, the Quest 2/3 is nearly as good for watching videos in bed.

There's no need to pay the extra $3,000 or whatever it is.

I'm very curious to see if more people adopt the AVP for productivity while traveling though. That's what the Quest seems pretty useless for still.

I have both Quest 3 and AVP, and they are not even close for “watching videos in bed”. Partly just pure video and audio quality, where the AVP is a generation ahead, at least.

But also for less than singleminded video watching: if you get a text, the notification appears. If you are watching a long-running deployment, you can have a terminal window off to the side to glance at periodically.

And if you want a drink of water or even to get take the glass to the kitchen for a refill, you don’t take the headset off. The passthrough is that good on AVP, where Q3 is sort of functional but very acid trippy with bending geometry when you look around.

Is AVP “worth” 5x as much? IDK. Depends on your income and priorities. But it is certainly a dramatically different experience.

(Q3 is still hands down better for games, BTW)

Apple consistently has the advantage of good UX: tight integration with their ecosystem, including message notifications and calls. Costing around $3,000, the product must feel solid with top quality components. I'm wondering about the external battery and how it affects mobility. I imagine there might be downsides to having to carry it around, but maybe it's not as bad?

The battery is annoying but not terrible. At least as a guy who wears pants with pockets, it’s not much different than having a cell phone in a pocket.

The cable is also somewhat irritating, mostly in that “a big just landed on me” moment when an arm brushes it.

It is definitely suboptimal and contributes to the AVP’s vibe of absolutely incredible and yet not quite ready for mass market.

i've definitely snagged the cable on a doorknob walking around the house, it's not great, i've been meaning to get one of those ipod armbands people used to wear at the gym to avoid the issue

Ha! This morning, laying in bed, I put on my AVP and saw that Dune 2 is now on Max. "I'll just watch five minutes..."

You can imagine what happened :-)

No one for whom money is still an object should buy one, but absolutely everyone should schedule a demo at the Apple Store. It’s heavy, uncomfortable and has few legitimate use cases, but the demo is incredible. It’s a glimpse of what’s to come.

Whatever happened to the Vision Pro? It seems quite strange for Apple to have flubbed its release so hard and to have also gotten away with it.

bunch of people tried it and returned it, apple didn't really say anything and kept shipping OS updates

developers are developing but it's a small clique

i don't think it's regarded as a commercial flop because there's bound to be a gen 2, hopefully by then there are some apps in the app store

but i agree with your characterization that they got away with a flubbed release