Could we make the web more immersive using a simple optical illusion? | Hacker News Reader

Could we make the web more immersive using a simple optical illusion?

shopify.github.io

685 pointsguiambros266 comments

Can't help but notice that we keep wanting to make content that should be flat more 3D (e.g., animated 3D backgrounds, scroll filler on corporate pages, the entire metaverse), and content that could benefit from being 3D more flat (like UI buttons losing their pseudo-3D bevels and not looking like interactive widgets at all).

Re UI, my theory is that this is partly because minimalism doesn't become dated as fast.

Related, I remember that what blew me away upon seeing the 3DS for the first time was not the 3d playfield (which looked exactly as I expected), but the UI in 3D. It really felt like a glimpse into an exciting future.

Here's why minimalism has such staying power. Fundamentally, pleasing design is about repetition and consistency. Brian loves patterns, that's all there is to it. Good graphic appeal isn't a seasoning like salt which you can just sprinkle on by paying for a few better "designed" assets. Its possible to make really pleasing design out of parts that are individually seen as cheap and low quality. Its also possible to make tacky tripe out of great quality parts. The important thing is consistency and arrangement. A pleasing design is one that lends itself to file compression.

Minimalism is a very easy style to pull off because the fewer elements there are the easier it is to make them all consistent, uniform, simple. It takes the least time to get it right. It can be handed off to someone else without explaining any "vision" or guidelines.

None of this is to say it is bad. Minimalism is good actually. Just that minimalism isn't uniquely good. Other designs can be timeless, they just rarely are.

I don't think we'll be seeing a move away from minimalism for the foreseeable future. The style itself will continue to evolve (hey, we just got rounded corners back!), but it'll within the confine of CSS so that no graphical/svg assets will need to be created.

We'll have minimalism until a browser maker decides to make background: golfstream-leather('brown') and felt-cloth('green') work in CSS.

I would assume this to be possible with CSS houdini, which extends CSS with custom JavaScript.

I say this as a lover of Flash - This was exactly why when people picture "Flash websites" in their mind, they think of these fucked up textured insane things with interface bunnies strewn around a red pleather couch or something.

A recent trend seems to be to combine minimalism, tackyness and deliberate inconsistencies in order to feel more playful and friendly. IMO these designs are also clear, because they allow themselves to let things pop out and have very strong accents and contrast.

Neo brutalism?

Brian may love patterns and minimalism, but I am Ryan and I love chaos and maximalism.

What if I told you Ryan and Brain were the same character this entire time.

Pixes music starts playing as buildings explode

I don't know what's going on, but I'm 1000% here for this Fight Club/Pinky and the Brain crossover. If it's your first time, narf!

See, me personally, this is what I want in my medical device software. Gimme the data. I'll learn the six different modes for each button, and fly through that shit to find what I want. Sadly, we're training people to live with less control and a shallower understanding by presenting everything in these slim little interfaces. Minimalism elides.

> See, me personally, this is what I want in my medical device software.

Minimalism is a major factor for cognitive accessibility. If you build for the lowest common denominator then of course qualities like control must necessarily take a dive. If Google gave you more controls for search then it'd be better search, but then it wouldn't be as minimal anymore.

On software you have lots of room to hide complexity for the pro user, but that's not the case for hardware.

Hah. There's truth to that, but just on a tangent, it got me thinking about the hardware of my youth versus the hardware of today. Like, when I was 8, the coolest thing to me was calculator watches. They had either 3 or 4 rows of rubber buttons, or the fancy ones had flat buttons. They weren't trying to hide their interface under a flat screen, right? They were going out of their way to give you lots of stuff to play with. They had different modes; you could cycle through stopwatches and alarms and little 1-line memos and addresses. They encouraged you to read the owner's manual that came with them.

Those systems taught people how to use them, and didn't necessarily sacrifice everything to a series of wizard screens. I think somewhere around the point where design decisions started being made to help onboard people into a piece of software or hardware faster, we lost the joy of digging deep in the user manual and finding cool shit.

That Casio calculator watch is still sold—I saw one in Walmart yesterday. I was also admiring a digital Timex Ironman watch that had an "OCCASNS" function, including customizable calendar and time-of-day alerts, which I had never seen before on those watches. It only took me a minute or two to figure out how it worked, even with a segmented LCD and odd contractions to fit the words on the screen.

(I settled on a classic analogue Timex with a leather strap. Can't beat them, really.)

I'm going to pull a print/graphic designer card and say that "minimalism" isn't actually a style. I think I understand how it's being used in this conversation as a shorthand for "2002-2012 Google UI", which had a massive impact on the way people thought about web design. But I don't think it's fair to say that the reason for the staying power of that particular type of minimalist design is simple repetition or consistency. I think it's a combination of laziness on the part of the designers and training of the population, and it has led to a lack of curiosity on the part of users when they're confronted with more complex interfaces. I think it's important to continue to buck that trend. Every interface should feel like a jewel box full of easter eggs. But having said that, design's purpose is to download information into human brains, and so [edit: clarity] the minimal opacity of the underlying information has to be the primary design goal in all cases.

Sometimes though, "minimalism" in the Google web design sense actually makes the information much too opaque. The ideal design requirement should be specified as that which gives the user all the tools to navigate immediately. If those tools are a little complicated, in the words of Thomas Pynchon, "Why should things be easy to understand?" This is where I have a huge problem with iOS, just for example, hiding scrollbars on components that don't obviously look like they're scrollable - so users never realize that there are more options off to the right. That's minimalist, but it's less than the minimum information you need to use the software.

Saying that designers are lazy and its their fault is a weak argument. For sure not all of them are lazy. Most of the time they are told what to produce.

What you are talking about is economics. Its possible to deliver good results faster/cheaper and the result is more timeless. Also clean interfaces are putting function and content in front. Thats why companies choose this.

So dont blame the designer but the boss/manager. Its like saying that app is slow because devs are lazy.

I'm sympathetic to working in big corporate structures as a designer where diktats you disagree with come down from on high. So I'm not blaming individual designers here. But I do blame art departments for not fighting back against non-artistic managerial marketing decisions, and some of that blame rests on art directors for capitulating, and some of it rests on designers for not making their opinions heard to the art directors. From a corporate standpoint, listening to your marketing team instead of your design department - and instead of having an open door to designers to walk in with ideas - is extremely limiting. It's not making the most of your resources. But design departments get a certain kind of laziness in their culture once it's the case that no one upstairs is going to listen, and this becomes a self-fulfilling prophecy.

I think we agree on a lot of things which aren't vocabulary. Well, that and jewel box full of Easter eggs. I have no idea wth you mean or why non designers would care. But you've touched upon one of my biggest gripes, the difference between simplification and reduction. A simplification organizes information into useful patterns so that it can hide away redundancy. A reduction cuts information for the sake of presentation. Reduction can feel like simplification because the end result is a nice crisp aesthetic, but the difference is apparent when you try to use it. Its like the difference between tidying your room and throwing all your stuff out.

My go to case and recent obsession is the NYC subway map. Here is the 1970s Vignelli map. https://mymodernmet.com/wp/wp-content/uploads/2021/01/massim... Here is the map that has been in use for the last 40+ years. https://new.mta.info/map/5256

Designers tend to love the Vignelli map, but everyone in the city hated it and it was short lived. To this day you can still hear condescending remarks like the Vignelli map was "to abstract for NY". "NY choose the worse map". But its exactly like you said, first and foremost the point of design is to download information into brains, and by that metric the current map is far superior which is why it won.

Consider the background geography. Ardent supporters of the London Tube / Vignelli style insist surface streets are clutter full stop. Maybe that reduction of information makes sense in London and Europe where streets are a mess anyway, but in a grid city that reduction doesn't make sense. The present map simplifies the streets down to just a few representative examples so you know the local orientation of the grid and have a few reference points. Simplification, not reduction. The information is still there.

The coloring system is another massive example of simplification. The Vignelli map seems to hate information compression, insisting every line have its own color and own dots. The current map reduced clutter by organizing lines by where they run in Manhattan, denoting the express/local difference with black/white dots. Personally I think that symbol could be less arbitrary, but most people have no trouble figuring it out quickly. Its a simplification down to a single line which makes everything fit in a tight space while being easy to follow. There have been changes in service patterns which have ruined parts of the current map, but the very original version had some nice symmetry to it (and mostly still does). Even though the rule is technically "color by Avenue in Manhattan", that rule lined up with certain repeating "color zones", eg both South Bronx and Flatbush being Green/Red, Upper West side and New Lots being Blue/Red etc. You could sort of count on certain colors to be associated with certain broadly defined endpoints.

Perhaps, this example, being so utterly detached from the modern web design world, is one you might use simply because no one has any personal stake in it. Its a lesson designers should learn more often. Hiding and removing != simplifying. Good != pretty. Organize information well and its often close to pretty on its own.

(I have plenty of gripes with the present map but that's for another time)

> Designers tend to love the Vignelli map, but everyone in the city hated it

Those designers were/are more illustrators than designers [1]. Design is, or at least should be, as much about the function of an object as about it's looks. A design that hinders functionality isn't a good design.

[1] I call the modern breed of such designers "Dribbble-driven"

Just want to say, I loved clicking on that mta.info link and seeing the SVG load in gradually, first the coarse map elements, then the coloured subway lines, then finer and finer details and labels. Felt like some kind of crescendo of dataviz, as if the THX sound effect ought to play over it. 10/10.

What gets me is the death of decorations, and then trying to guess what interaction you're going to get if you click on a certain flat color bit of a background window.

I keep thinking back to the winxp to 7 era (along with the various windowblinds/uxstyle third party skins), plus various furniture, fittings or tools/utensils/appliances from decades/centuries past (so antiques essentially). The physical items would have textures from the materials used and the processes to make them, maybe they'd be gnurled to add a grip, but also elaborations that aren't purely functional or just to make it look nicer. When you're familiar with those visuals they do seem to help you know what part of it you're looking at, and how it's going to act if you grab/click any given point.

Scrollbars are now an annoyance (when they're not hidden) as they're often low contrast and made to fade into the background, just another dull rectangle in a sea of others, rather than a distinct bit of UI chrome. There used to be a unique inline and small grooves at the centre point as I remember.

I'm not about to declare having the pendulum swing to the other extreme would be better, but it'd be interesting to see it explored again.

> Fundamentally, pleasing design is about repetition and consistency.

Fundamentally, what makes design pleasing is super hard to reason about.

If it was mostly about repetition and consistency, any webpage that used bootstrap or no css at all would be pleasing. If pleasing minimalism was very easy to pull off, nobody would take note of Apple products.

Apple is basically Bauhaus for nerd stuff - and that is no small thing. Perhaps if the microcomputer had been first commercialised in Europe they'd all look like that anyway.

Personally, I view Apple more as a fashion house that sells tech.

Those fashion designers make a pretty mean little RISC chip.

If instead of "very easy" would you be happy if I said minimalism is the "least hard" to pull off?

Cue Kevin McCloud from Grand Design whose standard rant about minimalism is that it’s actually the hardest to pull off because you aren’t hiding behind anything. All of your decisions are exposed, so you have to be even more thoughtful about attaining cohesion in the final form. Of course, he is talking about buildings which is a different thing, but I think some of what he says applies to other design fields too, including UI design.

> Brian loves patterns, that's all there is to it.

Who is this Brian you are speaking of?

The name's Damage. Brian Damage

Minimal "flat" UIs are possible for normal people to implement. In the 2000s I had no idea how I was supposed to make a button that looks like it's made of drop shadows and fine Corinthian leather for a project in my free time.

(For that matter I don't know how I'd get it done for a professional project either.)

Why not use the system default controls and not reïnvent the wheel?

Because nobody else did, including the system.

I was right there with you. I was very supportive of the improving app UI standards at the time but it started to feel like a contest of whose design has more visual complexity and nuance vs the real usability.

Windows Phone in particular was a huge breath of fresh air for me (rip) and Material Design a welcome evolution that I still feel strikes a decent balance.

> Re UI, my theory is that this is partly because minimalism doesn't become dated as fast.

Nothing looks as dated as "futuristic" stuff.

Minimalism is helluva easier in dev. Throwing a few solid filled rectangles is sure cheaper than painfully drawing skeumorphic features.

The 3DS didn’t do anything for me and a large part of the population. The same goes for the 3D glasses that went out of style years ago. The main thing it did was give me and others a headache. Otherwise, this wouldn’t have been a fad.

Other than this demo, the only 3D effects that actually work (ie 3D movies) are in VR

The stability of the 3D feature was greatly improved on the New Nintendo 3DS, which added eye-tracking using the front-facing camera so that you could tilt and turn it and you wouldn't see a double image. I've never personally heard of anyone ever getting a headache from it, and it would be a bit silly not to use it because it doubles the screen resolution and makes everything much sharper.

Exactly, it violates The Toilet Principle, which states that most of the time, the user of your app wants to do the thing they came to do, then get on with their life.

Wait until you see a 3D Immersive Toilet...

Imagine, for example, opening your favorite brand’s website and being presented with a miniature virtual storefront. You could look at their most popular products as if you were standing on a sidewalk peering into their shop.

I imagined this as instructed as was simultaneously bored and depressed.

Why does it always have to be about shopping.

I imagine a youtube where the ads would pause as soon as you don't look at them anymore. Like that one black mirror episode.

But yes, first promote some gimmicky fun stuff to get the foot in the door and when it has arrived, show the real face.

You’ve heard of YouTube Red? Now try A Youtube Orange

"Please drink a verification can"

https://imgur.com/dgGvgKF

"McDonald's!"

https://mobile.twitter.com/TheStreet/status/1612888601196433...

The other way around, for sure, to distract and make you look back at the thing moving in the corner of your eye.

They mean that the ad won't finish until you watch it.

> Why does it always have to be about shopping.

It's published by Shopify, so in this case it's because it likely would not have gotten published otherwise. Unfortunately E-commerce companies are not known for their "research for the fun of research".

Disagree, VR stuff at Shopify really strikes me and a lot of folks as a pet project with unlikely business outcome but they still do it for the kick of it.

How so? Their VR and the Web page on their website has mention of products in the first line?

https://www.shopify.com/partners/blog/building-a-vr-shopping...

Because it's the web, and that's all the web is about.

It's a lot of what life's about though too, so perhaps it's justified?

Justified in the sense that we should accept that there is no other purpose to life or existence then to shop?

For every article about shopping we would be seeing tens of thousands of articles about breathing then.

Or, imagine your head tracking data is being uploaded and sold to the hugest bidder! They could tell whether you have a health condition which they could sell you an expensive drug for! Or raise your insurance rates! Wow this IS exciting.

The smart watches already provide a lot more data such as heart rate, blood oxygen levels, and activity monitor to name a few things. However, it’s just a part of the little data collector that nearly everyone has in their pocket.

My point is we’re already there and we’ve been here for years now. If laws like the GPRD passed, then it should be possible to counter or neutralize the problem

The blog we are reading right now is Shopify's blog, so I have a theory.

Another interesting question is "Why is shopify's blog the place talking about this, instead of somewhere else?" I remember that demo... it was amazing and then it did go nowhere. With modern face recognition and tracking, it should be entirely possible to use it for some nifty no-touch experiences.

The blog we are reading right now is Shopify's blog

tbh I gave up reading after the bit I quoted and hadnt actually clocked it was a Shopify blog (didnt spot the little icon in the corner or look at the url)

Still, I stand by my refrain "Why does it always have to be about shopping" as a refrain that is rarely out of place

Well, it made into Amazon Fire Phone, and then went nowhere.

It's a relatively invasive technique that requires a computer's camera to record details of a person's face, and that's marketable data that any commercial shopping site would likely collect and sell on to other parties. It'd be much simpler to implement arrow key / spacebar navigation through a 3D scene (still computationally expensive relative to rendering the standard HTML/CSS/JS page), but then that data couldn't be collected.

In general interactive 3D on the web still uses too much computational power on the client-side for most devices.

> invasive technique

It wouldn't be so bad if it were a browser feature, i.e. a website can use an "Eye Tracking" API instead of the Camera API (with permission granted), and get limited data.

But of course, just like gait detection and etc... there would be a whole genre of preprints dedicated to eye tracking fingerprinting

"Details of a person's face is marketable data".

Okay, it's not that I don't believe you, but what do they do with that data? How does it help? Unless you're marketing acne cream, I suppose.

Uniquely and reliably track a single individual’s web browsing habits.

Is that not the holy grail of adtech?

People who fit into <some racial/ethnic/age/gender bucket> spend more time looking at <x products>, fed to advertisers who can tie it to swathes of your other online activity

Even if it’s _theoretically_ coarse data/bucketing, that adds up to a lot of data to help deanonymization and targeting. Even if that’s done in the spirit of “just for ads”, it’s not a big leap to political targeting etc, and with the right data leaks, potentially even actual harassment.

Some bad actor would check out what is in your apartment and then market to you.

Say they see you have legos, they market legos.

Or they see what you have (say a gun) so they rob you.

Or they track your habits. We noticed your apartment is dirty/messy - pay us a ransom or we post on linkedin or call CPS. Or use our cleaning service.

Also I bet you have to walk fully dressed when browsing (especially women) and you need to shave, or your social score drops.

Ah and will you know if the website ever stopped watching? Is it watching you (and recording) having sex?

Knowing exactly where the user is looking is pretty good data I would say.

"User looked at the buy button 5 times in the session but they did not buy" and so on.

Couldn’t this sort of thing be built into the browser which could then just expose an API for head orientation and eye position without any privacy invading details?

Physically you'd still need to have the camera on, and having the "your webcam is in use" LED become ever-presently on defeats the ability to detect when the camera is actually being used as a camera.

Imagine the possibilities! If you have enough facial expressions, looked for n ms, moved head n yards, with enough a-b testing you could dynamically design products. Like Weird Science only worse.

An awful lot of us already wear glasses, whether prescription or blue light blocking.

If tiny IR lights or something else distinctive that the camera could recognize on the glasses could be used instead of actual eye/face tracking, I'll bet even more people would wear them.

It is not imaging your face like a normal camera would. It is using an infrared camera to pick out the location of two dots (the relative position and distance of the two dots in the camera”s field of view). That information stream could be compressed to a few bytes per second without losing tracking fidelity. That does not seem very invasive. Even if a high resolution pixel stream were to be sent from the camera to an untrustworthy, persistence layer, it would be infrared, which doesn’t seem very privacy-invasive.

Because this is from someone who works for Shopify and has to justify their 20% project to their bosses.

Why can’t people have a little more imagination and see this as a great new data visualization UX like for things like charts or maps, instead of automatically panning it to death?

This could be very useful in hospitals.

CAD and 3D modeling software?

It has to be about shopping because pr0n actually drives new tech and that venue would not be pr0n friendly.

I would imagine this would be an obvious easy sell for CAD, so there must be something wrong with it, perhaps a patent, holding back progress. I will say that I've done CAD for projects (woodworking and making STL models for 3d printer) and people who do not CAD think that CAD people look at 3d animations 99% of the time, but IRL when I do CAD I spend most of my time thinking very abstractly (how to balance the spacing of power supply and PCB such that it looks nice, is easy to access, thermal concerns, wire routing... Or I can look up the clearance hole dia for a 4-40 screw now what is a common drill size with sane tolerances to drill it, how about an eighth of an inch?)

> Why does it always have to be about shopping.

the subdomain of this submission offers a clue about that

Here on the internet, there's nearly three things to do!

https://youtu.be/G9FGgwCQ22w?t=183

I don’t know any of the science behind it, but my experience with these kinds of ideas (including the 2007 wiimote target demo which I replicated and showed to family), is that they feel exhausting to use beyond a 15 min novelty.

My guess is that the brain detects that the motions and such are not quite right, and like a wrong eyeglass prescription, your brain and eyes work overtime trying to figure out compensation.

I had the same thing with compiz when it first came out. I loved it in all the videos; and the first day, but I quickly switched back to plain old desktops.

I actually did use Compiz+Beryll for a fairly long while. Admittedly, the bugs were annoying, but after switching off the most gimmicky bits or at least making them less pronounced (like Wobbly Windows, lol) I found it to be quite nice to use. It felt a bit like if you were using WindowBlinds but supercharged.

I later switched to i3wm and threw away desktop effects though. I wouldn't mind them, but I don't miss them either.

Oh wow, wobbly windows. I miss them.

https://youtu.be/nbCg9_YgKgM

For a blast from the past

I still use (KDE Plasma's) wobbly windows, but set very low. In general in GUIs I prefer a very quick animation to an instant change, perhaps because it helps make things seem more fluid and dynamic and because it guides my eyes to where a change is happening in case I'm not already looking straight at it.

Oh good times. With Firefox 2 (or maybe 1.5?) no less!

I remember spending a lot of time trying to get Compiz to work. The rain effect was my favorite, it even had windshield cleaner!

Edit: here's the rain effect - https://youtube.com/watch?v=1ZKmQS_MNAg

The sole reason I had compiz back then was to have the cube effect. I do miss my cube even in i3

I totally agree: I was a big fan of the desktop cube. I think I'm okay now, as I think I'd just about die without at least numbered workspaces, but I might be OK with getting that one back.

This is starting to happen to me with bad AI art. Subtle things that I didn’t notice for month now look glaringly out of place.

could a (much) bigger screen compensate for this?

I assume one of the biggest problems is, that both eyes see the exact same image while in a „real 3D environment“ both eyes would see slightly different views of the world where the brain then reconstructs the depth out of it. Here we have the „mono“ version with some „fake stereo“. It works surprisingly well, but the brain calls the bluff.

No you can achieve a very comfortable effect with a very small field of view, the difference is due to inaccuracies in positioning resulting from not having a close enough approximation of the eye's position or too low of a frame rate so the image is lagging behind motion. All of this stuff has been very actively developed by the VR community.

The thing is with the wiimote (atleast with motion plus) and the switch, nintendo really tuned in the motion. I think you can give anyone age 5-99 a wiimote (or a joycon), say play bowling or tennis, and they can pick it up within a few second

Barely anyone else puts the effort into getting gestures right. Part of that is the developers spend so long developing the gestures, they don't know whats natural and whats not anymore.

Big example is windows precision drivers. In theory they should be more natural, in practice they are terrible.

Agreed. I disabled that iOS background effect where it moves as you tilt your device. It was interesting for the first couple times, then annoying.

> is that they feel exhausting to use beyond a 15 min novelty.

Anecdote aside, would this potentially change if it was done for something with utility like data visualization? Unlike a VR headset or the original demo, you don’t need to something radically different to experience something radically different ie holding or wearing an IR light. You just start using it with your existing hardware

> Imagine, for example, opening your favorite brand’s website and being presented with a miniature virtual storefront. You could look at their most popular products as if you were standing on a sidewalk peering into their shop.

I am imagining it, and it's worse than the current norm.

Storefront branding using "smart" displays has been a variation on the same regurgitated idea since 1908[1].

[1]https://en.wikipedia.org/wiki/Selfridges#Windows

Physical stores already are in 3D, the internet is a different medium that actually is composed of multiple mediums, depending on what device you're accessing the content from.

This is exactly why I think 3D TVs died out a couple years ago: once the novelty factor fades away, you realize that it's actually hindering your initial goal.

Nah, it was because the costs of the content were always significantly higher than other content.

Besides the cost, they died because for a large portion of the population and a large portion of the supposed 3D media, the 3D effect didn’t work. It just gave people headaches and eye strain.

Not to mention all of them being locked to 60 Hz input while proudly advertising 600hz refresh rates (mostly plasma). Literal dark ages

> Imagine, for example, opening your favorite brand’s website

My what?

Do these people ever leave their fantasy land?

Cynical take: Seems like an excuse for Shopify to add eye tracking to product pages for ad analytics

Unfortunately it's stuff like this that's poisoned otherwise cool new tech for me. I can't help but wonder "how will this be used to advertise to me?"

One of the cooler takes of Bitcoin or "crypto" is that you can use native tokens to pay for visiting websites. A browser plugin combined with a lightning wallet for instance could enable a website to request a payment of any amount of Satoshis to view the site, or it will redirect you to a 402 error page. This generates a way of monetizing content without the need for ads or tracking.

But then you need something like Nano instead of Bitcoin, which has 0 fees. Ideal for such micro transactions.

This is implemented in the Brave browser.

I'm not sure something that actively incentivises writers blather on endlessly, bury the lede, split their content over dozens of pages, slightly raising the price knowing that people will become densensitised to reading your per page costs, lock final review scores behind more expensive paywalls than the lower priced intro pages, and who knows what other dark patterns will be invented all while forcing me to make a determination every fifteen seconds on "is this worth the price they just asked?" is supposed to be a cool thing.

If you thought recipe articles were terrible today, just wait until you have to pay 0.01 for their life story and then 0.03 for the ingredients only to discover that it will cost you another 0.15 for the actual instructions.

Micropayments existed long before bitcoin and adding a blockchain mostly just makes them more awkward.

Crypto in general yes, but the lightning network specifically no. It's utterly impractical for most users, unless you use a third party hosted wallet, in which case almost all the benefits of crypto, like anti-lock-in and jurisdiction-agnoticism, go away.

Crypto-powered micropayments could potentially massively boost revenue for content creators, including socially valuable platforms such as news sites.

Bitcoin has a throughput of 1 KB/s for the entire world, so it will never be used for micro transactions.

Or invade my privacy otherwise

Yes, it'll be used for workplace monitoring to generate easy to measure and rank KPIs for review time.

Research indicates the ideal reading speed for corporate PR emails is 40 wpm and your teamwork KPI at review time will be stack ranked based on how close you move your eyes to the ideal 40 wpm reading rate.

> I can't help but wonder "how will this be used to advertise to me?"

Or lie to you? VR storefront eh? Bet the product will look a lot better than it actually does in real life.

If the VR is good enough, why would you even bother to visit it in real life?

To see the actual product they're trying to sell?

This is what TFA quoted as an application for the tech, virtual storefronts presumably for physical products.

I mean, that's true today. I'm sure you've seen plenty of shops egregiously misrepresenting the sizes of products via forced perspective or outright editing in images. If they were to upload a 3D model it would only get more difficult to mislead about the physical characteristics of the product.

You think theyd Upload it unedited? :)

I'd imagine not many people are willing to give the webcam permissions to e-commerce shops.

They will during the next lockdown when its give permission or don't order food online to eat.

> Look directly at the captcha, please read in a clear voice while drinking verification can.

> Whoops, looks like you may have blinked or looked away! Please try again. If you have run out of verification cans, you can say "I love Cosco" to temporarily credit your account with one VCT while we dispatch a new box to your address.

I think it's important to remember that Shopify doesn't have a singular ad product. It's a platform to build a web store on, and merchants can add whatever they want to their shop, but Shopify doesn't collect any data across shops.

That's the shop's data, not Shopify's.

Bias: my team at Shopify helps merchants launch their shop and get their initial sales.

Y'all are happy to spam people with onboarding messages at the email address that a customer gave you and you didn't bother to verify.

Does Shopify have a sales team? Does Shopify have a marketing team?

What's a more likely and more powerful pitch that the marketing team could use to better increase sales of the Shopify platform given this tool they've developed:

1. You can make your stores more FUN! With this little funny tool. 2. You can have more accurate tracking data on what your user is looking at at every moment when they're browsing your website's catalog.

Why not both, really? It's not in Shopify's interest to keep eye-tracking data, but it is in their interest to encourage people to use this feature because we all know this kind of data is a selling point for an e-commerce business.

You really sound like the gun manufacturer that, after a mass shooting, says it's not the people that are flooding the market with guns that are the problem but the individual that did the shooting. This discourse might resonate with some, but not me I guess.

I think it's disingenuous to compare an online store platform to a gun manufacturer.

What I'm saying is that individual stores can track their customers however they like, but that there isn't an overall tracking system across all stores.

Shopify's selling feature is that your data is your own. Your customer data is your own and not used by Shopify. That's the competitive advantage over Amazon, who will look at your sales data and use it to compete against you.

It would be penny-wise but pound-foolish to do what you suggest.

I don't think it's disingenuous? Gun manufacturers develop guns that gun stores buy to sell to people, Shopify develops tools to track people that they sell to business owners. The gun can be be bought by an individual that can do good or bad. Same for Shopify.

Shopify is developing tools that can be used to do harm or to do good. But when they develop tools they know will be used for surveillance, then they can't just wash their hands away. There is responsibility there. Shopify knows how people are using the tools they are creating.

I think facilitating independent shops is positive, but developing more and more tracking functionality to get these shops to be "data driven" is harmful to society. Although that seems to be "the market pull", a lot of times the most ethical choice is not necessarily the most profitable one. Shopify wants to compete with Amazon and they believe a good way of doing that is offering their customers (business owners) better ways to track their customers (shoppers).

All you're doing is giving a third party the gun and your justification seems to be "well if we don't do it, they'll get their guns from amazon anyway". Who owns the data matters, I agree, and I commend Shopify for that, but it's a problematic trend that is being facilitated by these tools anyway. I mean, we should get rid of the guns, not go "we need more good guys with guns". That's my opinion at least.

It's ironic that 15 years later, the latency between the movement of the camera and the targets has only got bigger (original video vs the first one in the "Final prototype" section).

First thing I've noticed as well - feels like the same 1990s VR experience.

To be fair, they're running it entirely in a web browser using only software with no specialized hardware.

And probably hasn't been optimized nearly as much as it could. It's clearly meant as a fun tech demo / early exploration.

This is super cool. I wish WebVR was still a thing and that it had gotten more love, better browser support and more libraries adept at handling head tracking, stereoscopy, etc. During that brief period where it was standard in Chrome, I happened to be bidding a bunch of interior architectural designs for a client, so I scrapped together a demo script in ThreeJS that let me stitch together prerendered stereo panoramic renderings, and insert some interactive 3D objects / animations inside the skylight space. I patched in support for the pointer remote from the original Oculus Rift, and just stuck our architectural renderings on a private website and mailed the client a couple Oculus Rifts as a gift to check them out. It was a big "wow" factor and landed us a lot more work.

None of this is to say that it would be a great idea to build websites this way. Websites are made to convey the desired information quickly, so unless the desired information is a full 3D understanding of a space I have a hard time seeing the utility of building out a 3D space as a metaphor for information. Did that make sense? Let me say it differently: We've tried a lot of visual metaphors in my life for organizing and exploring information spaces. As it turns out, the simpler ones are usually the most easily grasped and therefore the most efficient at doing their job (i.e., remaining "invisible" and serving as a transparent container for information). Even 3D interfaces that do serve the purpose of conveying spatial relationships and art often fall into the trap of trying to do too much. This is not meant as a paean to minimalism (at all - I love clever interfaces and I appreciate when information design makes you figure out a paradigm to get the most out of interacting with it - as long as "the most" is defined as more interesting views of the data I'm trying to get at). Just to say that design should showcase content, not itself, except in the case where the content is supposed to showcase the design, and then you can go nuts.

WebVR is still a thing with WebXR and at least the browser devs at Meta are pushing it forward.

The fire phone did this 7 years later with 4 front facing tracking cameras and nobody cared

The New Nintendo 3DS was based around this, and although they abandoned it I think there's still a lot of room for something like it. It's more AR than AR is.

Did it do head tracking?

It always seemed gimmicky to me, but I’ve heard that the Mario brothers 3D worked a lot better with the depth element than in any of the remakes (I did it enjoy it just fine on my switch)

The original 3DS didn't but the "New" one did eye tracking like this, and for both eyes since it had a 3D screen.

No, it relied on the fact that the player would ~always be looking straight at the screen at a relatively fixed distance. There was a slider you could use to set the strength of the depth effect, all the way to disabling it completely.

It was fun and worked pretty well, but it was very much a gimmick.

EDIT: Just saw the other comment, I didn't know the New 3DS had actual eye tracking. Neat!

I bought a new 3ds and it was absolute bliss. The mario bros 3d game was incredibly intuitive to play, that 3ds has spoiled any 3d platformer for me. I tried playing the new 3d game on the switch and even though it has shadows, I kept missing platforms etc.

I think the original 3ds's lack of tracking is what turned it off for most people, though.

The 3DS don’t do head tracking, it was a real 3D display that displayed a unique image to each of your eyes.

The New 3DS also did eye tracking, which is why I said it did.

nobody cared because you had to buy new hardware (the fire phone) to get it, at the expense of the current hardware you already really liked. In this case, you don’t have to get any new hardware.

Relevant video by DankPods: https://youtu.be/nJRYDbV7jr0

He managed to get a brand new in box one, not spoiling if it works or not you'll have to watch to find out...

A couple of 3D display products are coming this year: https://www.leiainc.com/newsroom/reservations-open-for-world... and https://arstechnica.com/gadgets/2023/01/asus-new-16-inch-wor.... As my username might suggest, I’m excited! I hope they become less niche with time.

Excellent. Thanks for those links.

Toshiba and HTC had this around 2011 when the mobile OSes were really bad with no updates available, phones produced by HW companies like Toshiba who had no idea on how to support sich HW software-wise. Even the Laptop 3D screen's drivers used to break with the next larger Windows update.

So I'm happy to see these devices return. I hadn't thought that this would happen anytime soon. However, theses are integrated devices again. Bit afraid that the support will be as short-lived as back then.

We haven't kept our FHD 3D TVs for the same absence of updated 3D TVs. But these have unified interfaces at least (Packed HMDI Stereo and SBS or TOB for normal HDMI). So maybe a standard external devices might make more sense than those integrated tablets and laptops.

Time to charge my Fuji Real3D cams once again, I guess.

> Imagine, for example, opening your favorite brand’s website and being presented with a miniature virtual storefront.

I would puke

TrackIR is existing and mature product using the original idea.

Sure it requires extra hardware, but it's amazing in first person games that support it. You get to use your good monitor, don't get dizzy like in VR and it could be quite cheap.

I think implementing similar thing with just camera and computer vision is a good idea, let's make this more common thing.

Open track with neural net tracker can do what you're asking. The default configuration is a bit wonky but it works very well after some smoothing here and there.

Benefit of trackir is computation being done partially in the device/camera (as opposed to diy clips, which I use for my flight sims).

There have recently been some cameras released that don’t require ir-clips and offload the computer some, but feels =\ having a webcam requiring constant camera and/or tracking.

But for games it’s great and already in use.

Is it though? My understanding was that TrackIR was nothing more than an IR webcam surrounded by IR LEDs, that also served as a dongle for the freely downloadable "driver" which contained all the tracking code. With OpenTrack, a webcam with the IR filter removed, and a 3d-printed clip, I get results indistinguishable from TrackIR.

A system that uses a visual light camera and no reflector clip will inevitably be slower, as it is a much harder computer vision task. But I don't think hardware acceleration comes into it anywhere.

The techniques themselves are very cool and inspiring.

That said, the reason such things do/don't take off isn't about how cool they are, generally. It's about how useful they are.

Even for games, immersion and realism are secondary to gameplay. For most applications, it's hard to see how a "realistic window" is useful. A flat paper, paper-like screen is better to read off, for example.

Cool tech. Definitely worth exploring gaming potential. Not easy to apply to other uses. There's no benefit to making an online shop window more like a real shop window. Catalogue is what you want in any case.

Just apply this to CAD or 3D modeling software.

I can also see doctors having a use for this tech

at first id think this was cool but moving my head constantly isnt that gonna start hurtin?

i think keeping the 3d AR aspect of it and moving the object/mesh with a very intuitive controller for 3d sculpting instead of having to turn my head and walk around behind it to modify it is the way to go.

We use this concept to great effect in a [CAVE] where we were limited to 2d projections. The thing that many are not aware of is that stereovision is just one of many ways we perceive depth: https://en.wikipedia.org/wiki/Depth_perception

[CAVE](https://de.wikipedia.org/wiki/Cave_Automatic_Virtual_Environ...)

Broadly speaking: yes! I wish we used this more, it's not hard and it looks very convincing and subtle stuff feels quite good...

... when latencies are VR-like. When it lags even a little, it feels awful. So it tends to work pretty well for tech demos like this, and not at all when exposed to the real world.

Specialized hardware and software stacks can do a pretty good job - see the Fire Phone for an example there. It did quite well on technical notes, the experience was smooth and battery-efficient(-enough) and the illusion was almost fast enough to be transparent.

But it failed. Unless we get this kind of thing baked into everything for years, it will continue to fail, because there will be no content. And integrated webcams and their related software continue to be utter shit. We're not close to (or even moving towards) being able to use this, unless Apple surprises everyone in their next lineup and kicks off a renaissance.

..... and that's before getting into privacy issues with exposing an API that reveals info about your face. It's a harder practical realm than it seems it should be, when people have been making very effective tech demos for decades on commodity (but specific) hardware.

Less is more. We don't need 3D except in a few cases. Computer output is about information, not trying to make things look like in real life 3D. When we read a book we just need some black-and-white text.

There are great applications for virtual reality for instance in medicine. But in general 3D is only needed when it is needed. If it is not needed then pushing it onto users becomes a distraction.

Medical VR [1] is a great example of that.

[1] https://www.medicalvr.eu/

I agree. Gaming is an obvious use case, as the article mentions. Head tracking is a great and simple way to increase immersion, without requiring VR and bulky headsets.

I'm surprised more games haven't experimented with this. On PC, a barrier might be knowing the camera parameters and screen dimensions, but those can be easily calibrated, as the article shows.

On consoles, however, we've had cameras for two generations now, and the technology has been mostly underutilized for gimmicky features. The only game I know that did something really interesting is Alien: Isolation on PS4. With the PS Camera, it could do head tracking and noise detection, for some very cool interactions.

The microphone is also severely underutilized. There are many neat applications, especially today with AI voice recognition.

> And even if all that becomes possible, will users be willing to close one eye to enjoy immersive experiences? That’s a question we can’t answer.

No.

Are there any phone apps that do this kind of head tracking using the front-facing camera? I imagine this would simplify the camera calibration setup since most phones have a known geometry relating the camera to the screen location, unlike most webcams

This plus tracking phone position plus ML vision to fake up 3D from the front facing camera could make for a very cool FaceTime “portal” experience

Amazon built an entire phone around the idea back in 2014. They had cameras on each corner of the phone to improve the effect.

https://www.theverge.com/2014/6/18/5821294/amazons-fire-phon...

The web is plenty immersive when there aren’t 84738473 ads and user hostile UX layouts and popups and shifting layouts and all the other things everyone ought to already know are bad.

The alternative is that nearly everything worthwhile is now behind a paywall, which is the reality now. HN audience can afford the paywall. However, large number of people in world cannot. Piracy is not a sustainable solution either.

Now we can all look forward to the day when websites refuse to load because they want camera permissions for fake depth, shiny buttons, and other random visual effects.

Also they'll analyze your face and send it to the ad tracking agency.

I have a somewhat related question/thought regarding daily desktop use: is it possible to have the sound from the speakers follow the browser or window that's playing it? For example, having a podcast playing on the top right corner while a different website on another browser tab also plays sound, but with the sound source corresponding to its location, making it easier to distinguish between the two. It would also be really helpful when you have a ton of browser tabs open be able to find it with a more physical metaphor.

glue together web audio spatializaton and the coordinates of the window and you've got it.

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_A...

That idea has been around for years, for "virtual windows". This is from 2010.[1] The illusion only works for one person at a time, so, as a window, it's not too useful. For games, as someone pointed out, it works fine.[2] Although it hasn't gone mainstream.

[1] https://www.youtube.com/watch?v=Vqu9NuINKbc

[2] https://www.trackir.com/

I replicated Jonny lees demo to see how it looks in person but soon learned the problem is one of optics. Without presenting the content at the right focal depth you don’t get past accommodation and binocular convergence conflicts. Without shutter or polarized glasses you also don’t get past binocular cues - so it looks great on a video where there is one camera view but once you add a second human eye it stops looking 3D.

I started a company to try to solve these problems without using head mounted displays and we developed 3D displays that work using an array of rectangular lenses and micro displays plus fresnel lens sheets. We adopted this approach to a new product category we called virtual skylights where we replicate the look and feel of an open sky with a distance sun above you. As you walk around a room the sun will follow you in the sky across units due to the optics and there is no need for active head tracking for this particular case. The effect is really amazing even though the content in this case is super simple. Hard to experience on a 2d screen but we have some videos of it here, in the reaction video you can see how people react to seeing it the first time.

https://www.innerscene.com/a7/videos

Hats off, not only for the product but also for pursuing and solving the problems that you mentioned! Really cool! Just curious, is the technology prohibitively costly for individuals? If so, do you have any plans to make it accessible to mainstream public?

Thanks! Yes - we are getting installed in a number of homes as well as businesses. We are still at the higher end of the market, but comparable in cost to a real skylight after install and maintenance cost - and you can put it in places where real windows or skylights are impossible. We are working on some pretty low cost models now and will have something this summer I think will work for most home owners. In some parts of the world we sell through partners who maintain their own pricing, so we don't have public pricing on the internet but I'm happy to give you some info if you shoot me an email at jonathan@innerscene.com or DM me here.

Has anyone considered the supply vs demand problem or perhaps there's a better way to phrase it?

Yes, that is a VERY attention grabbing presentation for consumers.

Why is it dead in the market?

Lets consider the SUPPLIERS.

I would love to buy antique used electronic gear off ebay with a full high res 3-d model of the actual device I'm buying. Lets say its a Tek 700 series oscilloscope because I love paying insane shipping fees, or maybe I'm just insane (note WRT this being an insane hobby, I need lead-silver solder, vacuum tube tester, etc). I need full photographic high res 3-d models of that particular scope including all its "problem areas" and partially replaced 'black-beauty' capacitors (or are those a HP-only thing?) and partially replaced leaky electrolytics and toasty looking resistors and all kinds of fascinating detail, including exterior shots, and all sub-chassis inside the machine.

The problem is IRL you're pragmatically honestly lucky to get a single real photo of the device you're buying; not uncommon for someone to copy a pix of someone elses device when selling you theirs. "well its close enough' and in a certain defocused abstract sense that's even true...

So ... we've solved the viewer-consumer side but who's going to generate the models for most realistic apps? Like consider clothing sales online. I need a new shirt for the gym, and its cool in an abstract sense that I could see "a model" wearing my future shirt, but I'd be pretty interested in the experience of seeing "me" in the shirt as opposed to the model.

Where is the data going to come from, if/when we have a way to display it? I think this is the business problem where the display tech is stuck...

For the intrinsic parameters of the webcam, that is a property of the hardware and not your room, yes? If so, it seems possible to crowdsourced that information to build a database. Then for most users, it's just a matter of entering your product model rather than taking 50 checkerboard photos.

It's still not convenient, but it's a much smaller hurdle to overcome.

Agreed. My first thought was - what percentage of the target market could you serve with coding to specs for MacBooks, which have deterministic hardware.

Or if the tuning parameters exist with some range, use a traversal algorithm to find them, similar to an eye exam: “A or B? [change parameter]. A or B? [change parameter]. Repeat”

No because:

> for the optical illusion to work, you must place the virtual camera at the position of one of your eyes and close the other one.

However:

> Note, however, that it’s still a really fun effect if you use the midpoint between the eyes and keep both eyes open. It just doesn’t pop the same way.

Here's a demo from about 3 years ago that does something similar to the experiment in the article: https://compassionate-murdock-82fdbb.netlify.app/

It's interesting to see that these face detection models have been getting a lot better over the past couple of years, making it easier to stable and cheap webcam-based tracking.

That said, if you look closely at the videos in the original article, there's some noticeable delay between the camera moving and the program responding. The original wii controller based prototype that this was based off doesn't suffer from this delay because it used infrared LEDs. It's annoying to have to place those somewhere near your face, but once you have them there the face tracking because basically instant.

Aside from the skepticality of the proposed product & business idea, on the technical side there is tremendous value from accessible, robust and portable eye tracking. During my undergrad I was fortunate enough to work at a neuroscience lab where we built a 3D Unity game interfaced with a joystick, ECoG, and eye tracker to study attention and learning. If nothing has changed, Tobii is still the global leader in eye tracking hardware [1]. They have a wide ranging of products to choose from depending on your budget.

[1] https://www.tobii.com/

This is the perfect example of a really cool solution desperately looking for a problem

If you watched the link, you'd see the 3d effect, so is it looking for a problem when 3d is the holy grail of av entertainment problem?

Not sure what makes you think I haven’t watched the video! I watched it in full. But I don’t agree that 3D at home is a “holy grail” - for starters I’ve not seen any clear evidence of market demand.

A flat screen is a problem if you’re designing a 3D object. The flat screen is also a problem if you’re trying to figure out what’s wrong with a patient with flat charts.

Imo the real problem is more a lack of imagination mixed with too much fear.

couple live web demos with similar approaches you can check (!need a webcam for this to work) https://cables.gl/p/nrv_-a https://cables.gl/p/2Zf0Hk

I believe you need immense precision of tracking for it to look decent, cause brain notice all the imperfections and gives you motion sickness

I’m the sort of person who loves a solution without a problem and will devote months or years to it for zero result.

Having it be targeted as in-browser, not to mention for shopping (?!) Is weird.

But it seems like an interesting technique. I wonder if it would be more applicable to an iphone app where the hardware is more consistent.

Shopify serves online vendors.

Sure. But that doesn't totally dissolve them from the proposed usecase being non sensical.

The final demo was so slow and unresponsive compared to the 2007 version

If you're curious what an implementation looks like: Charlie Gerard's article[1] shows how to accomplish a similar effect in-browser. It's pretty straight forward although not sure what performance looks like in large scenes.

I like the idea, but don't know if people are ready to grant webcam permissions to untrusted any websites.

1: https://charliegerard.dev/blog/interactive-frame-head-tracki...

Is the web being more immersive really something we want or need as a society? I'm hoping that my kids learn to reject it, even though it's been my career for 30 years.

Coming from Shopify, this feels like a ZIRP (zero interest rate project). i.e. projects that need not bear any fruit in the next many years. Snark aside, this looks very cool, and I definitely see use cases in gaming (and maybe in web home pages--assuming you could get the client to turn on the web cam for an unknown site).

In the article I see no mention of accelerometers. Can one get nearly the same effect on mobile merely using these and not using the camera?

Sure - there are 3d-effect backgrounds available currently. Apple offers this as an OS feature since iOS 7: Parallax / Perspective Zoom for wallpaper. https://www.wired.com/2013/09/parallax-and-the-ios-7-wallpap...

Adding in Selfie-cam eye tracking should be relatively doable too.

Could then be added to an app, a game, or to the UI layer of a phone.

There is no value in reading text and looking at flat images in 3D and there are significant reductions in usability.

The reality is the fixed flat scrollable page was a huge step up from the real world reading experience.

Reintroducing 3D in a 2D space in a real 3D world is beyond useless without a dramatic shift in what media is consumed. And there is no atomic element of media right now that might benefit other than Instagram adding lights-box size 3D image to its feed for ads.

Wait… oh no…

Not bad, but I’m a bigger fan of the AR portals we’ve seen all over Reddit and Instagram for the past year, the most recent of which featured a virtual portal on the beach that you could walk inside and explore. I’m pretty sure everyone here has already seen it. This would allow you to devote a spare room in your house, for example, for the virtual space (or storefront). Why we don’t already have this is very frustrating.

One potential life-changing application I haven't yet found for eye tracking is to train monocular people to align their blind eye properly. I don't know if it's possible, but if it is at least partially successful for a monocular person's simulated depth perception, it could improve their eye contact with others and benefit their social lives immensely.

Phones seem like a better candidate for this. The iPhone is already equipped with a fast and accurate depth scanning front camera.

Why not use a tensorflowjs pose estimation model? They mention MediaPipe Iris but didn't find any tensorflowjs model?? https://storage.googleapis.com/tfjs-models/demos/posenet/cam...

I'd really like to see this combined with photogrammetry, so that in video chat you could see the other person with this kind of 3D effect (and maybe also adjusting the eyes a little bit to allow for eye contact). Could be great for work meetings and online tutoring.

I think this has been achieved on Google's research project Starline: https://blog.google/technology/research/project-starline/

> It’s unbelievable how much it feels like the OP-1 field is floating out of the screen. It’s almost as if you could reach out and grab it.

I can certainly agree with that, it's a real solid brusspup[0]-level illusion... but I'm not sure what to do with that, practically speaking, beyond "oh that looks cool." It kind makes me wonder whether my standards for 'immersive' are significantly different than theirs.

[0] https://www.youtube.com/channel/UCeQEKFH31vvD-InkTGSvCrA

I once saw a digital pinball machine with this sort of head tracking and the effect was WILDLY effective.

Heres a similar system: https://youtu.be/Hfz82-EaOQA

There was at least TrackIR providing that back then https://en.m.wikipedia.org/wiki/TrackIR

I remember having tried on plane simulators back then and, even if it was cool as a technology, it wasn't really useful on a computer as you simply don't move your eyes and head enough. If you do, the screen is out of your field of vision. For this to work well, the screen would either need to be massive and curved or move as you move your head.

Why did Shopify of all companies do this? Who would give Random Joe’s Shopify Store the permission to use their webcam? This looks like an absolute waste of time and money and a very impractical thing to use in general.

There are niches in e-commerce based around specialized tools which require more permissions and user engagement than a basic store site. In some cases, such tools account for a significant portion of the total revenue. It's not surprising at all to me that Shopify would invest in this type of technology, and I would assume that other big players are doing the same.

What niches and what specialized tools are those?

I actually implemented this at the time because it looked so cool. Unfortunately it doesn't work nearly so well in real life as it does on a video.

I don't exactly recall why but I think it's just that there are so many cues that you're looking at a screen that your brain won't accept that it's really a 3D scene.

It's a bit like those painted or chalked perspective illusions. They look great in a photo or video. Not so much in real life.

(I don't think it's because of stereoscopy because I am stereoblind.)

It’s possible to just convert the iris model to onnx then to tfjs btw. I got 90% the way to doing it once but basically concluded it’s entirely feasible

Heh. This is funny.

I'm not saying you're wrong but don't fall prey to a common misconception: difficulty in a project is not always linear, and the first 90% might be very simple whereas the remaining 10% of the task might be the most difficult and time consuming of the project.

Tilt Five has a take on this. https://www.tiltfive.com/

I'm fairly convinced that the immediate potential of AR is tricks like this, which will whet people's appetite for more.

Maybe. But when Apple added that kind of parallax/motion to iPhone/iPad people flipped: "It makes me feel sick." Well, some people.

Did it use eye tracking or was it just using the tilt sensors?

Tilt.

I audibly exclaimed WTF when they walked towards the screen. I was not ready for that effect.

I think this technique would be revolutionary for CAD or 3D modeling software. It always feels clunky trying to click, drag, orbit, fly, etc your way around a viewport, when all you really want to do is quickly peek around a corner to get some perspective

Seems interesting especially for the gaming industry, but it seems like it'd be exceptionally difficult to handle the edgecases in the real world; for instance multiple pairs of eyeballs staring at the screen.

This, alongside various privacy concerns of eyeball tracking, will likely nip this technology in the bud.

I think the technology might be more portable if you use a somehow-worn camera looking at the screen. The screen could display a calibration pattern and the camera could analyse it. Maybe a phone app would work. I think on phones it might be possible to retrieve the intrinsic values of the camera.

There was a time when the "web" was text only, lightning fast, and not unrelentlessly trying to hijack my attention. For a truly unique experience (for some), let's try bringing that back and see how people feel about all that free time they get back.

> This is an important limitation of this technique that a lot of people don’t know about: for the optical illusion to work, you must place the virtual camera at the position of one of your eyes and close the other one.

Interesting thing with the targets demo - less so with the zombie - but I notice that I only needed to shut one eye for the first few seconds to make the targets "pop out", and after that it worked for ages with both my eyes open.

When I read the title, I was hoping for something related to static images that look different than they are.

Like, bouncing objects on grey surfaces, or dots that aren't there

I would love to see these illusions explored more and used to enrich UIs in some way.

Need web camera integrated into the monitor and segregated from the motherboard so that these kinds of features can be provided without the risk of the webcam being hijacked and used to spy on the user.

A bit depressing in some ways. This is a pattern I've seen repeatedly, especially in the last 10 years or so - people want to do something innovative but shoot themselves in the foot up front by insisting it has to be in the browser. Then they discover that browsers are incredibly hostile environments for doing innovation projects, and instead of concluding the obvious (we should do it with a desktop app) they either ship something far worse than their original vision, or decide it can't be done at all.

This one gives off especially Stockholm Syndrome-y vibes:

> You need the physical dimensions of your screen, which can’t be queried even in modern web browsers ... That’s precisely what we needed, but MediaPipe Iris isn’t exposed to JavaScript ... Despite our failure to ship this to a massive audience, we still think it’s important to take a step back and marvel at what’s currently possible in modern web browsers ... It’s hard not to be excited about the future of the web right now.

Seriously? Their project failed to replicate a demo from 2007 partly because browsers don't expose the APIs they needed or even basic data you could get easily from Windows 95, yet modern web browsers are a "marvel" and "it's hard not to be excited about the future of the web right now"? How is that exciting? How is any of this a marvel? It seems the opposite of exciting to me; it shows that browsers still, after 30 years of continuous development, cannot be used to push technical boundaries.

The sad thing is their project didn't have to fail. They could easily have shipped Lee's demo on Shopify sites by writing a small native app for Windows and Mac, then making it download if the user clicks "View stereoscopically". The app would have access to the screen geometry and additionally detailed data from the camera driver, which could then be used to look up the necessary data from a Shopify API where they have pre-calculated the metrics for the most popular brands. On Windows the number of clicks required would be identical to in a web browser, because whilst the user would have to click the download in their tray to start the 3D view, as a web app they'd have to click a permission button to grant webcam access anyway so it amounts to a very similar UX. (On macOS Chrome it'd take one further click, on macOS Safari it will extract a .app inside a zip for you, so the number of clicks is equivalent to Windows).

It's because of repeated frustrating experiences like this with teammates in my own prior research projects that I eventually took the time to write Hydraulic Conveyor, because part of the reason devs are so reluctant to go outside the browser is that the distribution DX is traditionally quite terrible. Conveyor makes massive improvements there. The goal is to have a DX similar to shipping a (static!) website and it gets very close to that indeed.

Still, even now that distribution is fixed there are other reasons why research teams dig themselves into web-shaped holes like this. Not all of them can be fixed with software:

1. Experience. According to the StackOverflow survey, half of all developers have been devs for less than 10 years. We can safely assume that only a tiny minority of them have ever written a native app for Windows or macOS. Also about 50% of devs "dread" working with C++, rising to 60% for C and a whopping 75% for Objective-C, so that's not really conducive to working with the Windows or Mac APIs (Swift and C# do better).

2. Disinterest in learning. Probably due to "resumé driven development"; we devs always have one eye on the next job and what skills will be attractive to future employers. This can cause appropriate tech to get stuck in a catch-22/local minima where people don't want to learn it because other employers aren't using it, so people don't use it, so people don't want to learn it, etc. The strongest engineers will figure out what tech is most appropriate for a project and learn it if necessary even if that skill may be a one-off for that employer, but especially for people earlier in their careers they may prefer to aggressively chase what's popular to unlock job-hop pay rises.

3. The (inaccurate) perception that people don't download desktop apps. The proliferation of popular Electron versions of websites and general apathy towards PWAs is perhaps starting to correct this but it's still an issue I see sometimes.

4. Ideological loyalty. People like the idea of community/collectively developed open source tech, for all sorts of valid reasons, but this can easily turn into blindness. A complete failure to implement a 15 year old tech demo despite using vastly more advanced technology is objectively not "exciting" or "a marvel", it's actually quite embarrassing for our whole industry. But if you're emotionally committed to seeing technological progress through the lens of an ever larger HTML5 spec and nothing else, then it may seem somehow like this is still progress.

In light of the current top comment thread [1], I'm glad the state of browser APIs is holding back this particular "innovation". And I wonder if the friction of distributing and installing desktop apps is actually a good thing, even though it's affecting my current (non-AccessKit) project. It means you have to be really sure that what you're doing deserves to exist outside the good constraints of the browser.

I'm not a fan of VR in general. Some of us aren't comfortable navigating the physical world, and we don't want an emulation of the physical world to infect the digital (or more accurately, hypertext) world that we love and consider our true home. As Jakob Nielsen said in 1998, the Internet can be better than reality [2].

[1]: for posterity: https://news.ycombinator.com/item?id=35020429

[2]: https://www.nngroup.com/articles/better-than-reality-a-funda...

I mean, sure, I personally wouldn't care for this feature, but that's because I'm a bloke and don't derive much joy from shopping. My girlfriend though absolutely loves window shopping. I think this sort of feature would be far more appealing to her than me.

For better or worse we can't know how popular this sort of thing would be, because their project failed for tech reasons before even getting to the point of doing market research or usability testing. That's the higher level issue here, I think. If they'd gone the desktop route they could have at least tried it and seen what people thought, then later looked for ways to shrink it down to fit inside the browser. By starting with that requirement as axiomatic they never even got as far as running a usability testing day.

To clarify, my objection isn't merely that I dislike that feature, but that I'm concerned about it becoming a fad that gets foisted on those of us who don't want it and possibly can't use it (due to accessibility issues) but still have to buy stuff, access educational/training content, etc.

It would be better, IMO, if the web could get back to having multiple browsers that are serious about acting as user agents for a variety of users. Then, freed of the expectation that the browser must render a pixel-perfect presentation of the UI as it was designed, some browsers could provide a VR interface to the lovely accessible hypertext that sites should be providing as their one and only version, while other browsers provide a TTS-optimized UI, a Braille-optimized UI, a UI optimized for large fonts, etc.

Put another way, I think the web community should redouble its efforts to achieve the things that Hixie apparently gave up on. https://news.ycombinator.com/item?id=34622514

Hey mike, I'm the Diego, the person who wrote this blog post and implemented the majority of it. Reading this comment, I feel the need to challenge some of your assumptions. I don't expect to change your mind, but here's my take on this:

- I have only been a developer for 7 years, and I don't have a degree in Software Engineering. I actually studied electrical engineering and upon graduating I landed my first job writing device drivers for graphics cards, which got me started writing software professionally.

- I did that for 3 years, and after discovering a passion for computer graphics I worked at Autodesk for 3 years on a desktop application called 3ds Max.

- After all those years working on that old and heavy desktop application I started seeing web-based applications like Figma as the right way to do things. Nothing beats the convenience of opening a link to launch an app, and of being able to launch an app on any device. That's why I joined Shopify's Spatial Commerce Team.

- My favorite programming language is C++, and the demos you see in the blog post are written in C++. I developed them as desktop applications, and compiled them to WebAssembly using Emscripten.

- I have never done "resume driven development." I studied computer graphics as a hobby for years because of a deep interest in it, and I was poorly paid all those years.

Thanks for your time.

Thank you for your reply and I apologise if you feel offended. I was trying to summarise many experiences I've had over the years in one place and for sure, it sounds like what I've seen before doesn't apply to you.

To give a couple of examples, I've had someone tell me that they'd rather buy every dev who signs up for their platform (for free) a new iPhone, rather than write a desktop app to let them manage their API keys. I also worked on Bitcoin for a while and had to warn people that their users would lose large sums of money in avoidable ways if they implemented their idea as a web app, I was ignored because I was assured nobody downloads desktop apps anymore, and then people lost large sums of money in exactly the ways predicted. Note that Bitcoin itself was a Windows-only desktop app! This sort of thing is what I was thinking of.

Still, I'd ask you to reconsider the conclusions of your project. Yes it would of course be convenient to have it all run as a web app, but the alternative you faced wasn't a tradeoff between something more convenient and something less, it was between something a bit less convenient (albeit same number of clicks in many cases) or, apparently, nothing at all.

You clearly have the skills to try again with an alternative approach that runs outside the browser. It would be a cool feature, and even if in the end you conclude without being in-browser it's not convenient enough, it would be interesting to learn how users reacted in usability studies. I spend my time trying to make the desktop app UX as fast and smooth as possible, and if Shopify are up for the challenge then I'd be happy to work with you guys to try and build the most optimal UX possible for this project. Not every EXE is Autodesk! This kind of problem is exactly why I quit my job to make a new product for shipping desktop apps. Projects like this one don't have to wither on the vine.

The article states it could be really improved by using the Iris model, but I believe the MediaPipe Iris model is embedded in the FaceMesh model. Please correct me if I'm wrong.

Iris is a separate model from face mesh, but it relies on face mesh for part of the pipeline

> The first step in the pipeline leverages MediaPipe Face Mesh, which generates a mesh of the approximate face geometry. From this mesh, we isolate the eye region in the original image for use in the subsequent iris tracking step. …

> Face Landmark Model

> The face landmark model is the same as in MediaPipe Face Mesh. You can also find more details in this paper.

> Iris Landmark Model

> The iris model takes an image patch of the eye region and estimates both the eye landmarks (along the eyelid) and iris landmarks (along this iris contour)…

https://google.github.io/mediapipe/solutions/iris

Let's hope Apple's webkit for their AR glasses incorporates the ability to easily look at 3D like objects from 2D pages with a simple tilt of your head.

This seems like the kind of thing the FaceID sensors in iPad and iPhones could achieve. I’ve been impressed with its use for face painting in Procreate for example.

Instead, we decided to eliminate all visual cues for actionability and context. The modern UX designers hiss and scream if you suggest something as banal as 3D

It doesn't work if 2 people are looking at the screen?

It works fine but the image will only appear 3D to the person wearing the sensor because the effect is entirely based on responding to the wearer's head movement.

It doesn’t even really work for two eyes if the viewer is close to the display. Close one eye to really nail the illusion.

Think of it as the perfect approach for pirates with no friends.

Even the pirates with friends likely do their online shopping solo. Like most of us do.

Perhaps the pixels could be directional [1] [2]. Not sure if vaporware or an actual product, can't find more details since the Delta airport video.

[1] https://www.misappliedsciences.com/home/technology.html

[2] https://www.youtube.com/watch?v=82eAnMpeNZ4

I still remember the first time I saw the Wii head tracking video. I had no idea anything like that was possible. I don't get why it didn't take off

Because it's a tough sell to put a camera right in front of people's TV. IIRC, the only product managed to do this is Portal TV from Facebook...

I actually tried something like this and talked to lots of potential users about this (during 2020 pandemic). But the willingness to pay for the camera is quite low.

I remember when the Xbox 360 was announced and the core idea is that you plugged all your media crap into it. Combined with Kinect, it screamed, “we just want to know what you’re doing.”

> drink your verification can

The article describes an important issue. For the effect to look right, you have to close one eye. I suppose if you had a stereoscopic display this would not be an issue

Getting the camera intrinsics problem is might be solvable with the dataset of intrinsics (assuming that the browser allows to get the camera model ).

You can pose estimation of these parameters as an optimization problem. https://sites.google.com/ttic.edu/self-sup-self-calib

If this requires looking at it with one eye only anyway, would an eyepatch with something like a QR code on it help calibrate the system live?

Oh man, I remember this YouTube video at the top from when it was new. There was a lot of cool uses for the Wii remote when it first came out.

God, someone _please_ make the web more immersive!

The end product looked just like what putting on my TrackIR headset and booting up DCS World is like. :) Nice write up nonetheless.

It may work for platform like ipad that have limited number of configurations. Not sure though that Apple will be open to it.

This must be one of those new definitions of the word "simple" that I haven't previously encountered.

It’s curious that this post came a group in Shopify called the “Spatial Commerce Team”. I’ve often wondered why online shopping hasn’t evolved beyond galleries of thumbnail images. Today’s graphics should be able to simulate the immersive experience of browsing through a physical store—walking past shelves and riffling through merchandise.

I'd much rather scroll through thumbnails

I think you have it backwards. Shopping evolved from the practical limitations of physical shelves, to the more convenient non-physical, searchable, browsable online store categories. There is no demand for "virtual shelves" among shoppers.

How hard would it be to implement for a more popular webcam, like the one on a Macbook?

I briefly used a trial of an eye tracking mode for macOS. It was like having telepathy

Sure. Get rid of marketing and advertising and then we can start to talk about it.

Ah yes, an excuse to get consumers to adopt eye tracking at scale.

The original demo looks much more responsive

The latency makes it feel wobbly.

TL;DR: the parallax effect only looks 'impressive' on video, because our brain inteprets it as part of the filmed environment, but when you sit directly in front of the screen the illusion falls apart (unless you use some stereoscopic optics to trick your brain again into intepreting the parallax effect as 3D).

It was the perfect nerd-bait for YouTube, but not much more ;)

No.

But, really, the answer to that question is "no". The illusion simply doesn't work if you have two eyes and depth perception. That's why the 3DS needs to have a stereoscopic screen on top of eye tracking, or why VR headsets have 2 screens.

Immersion isn't really a problem. When I buy stuff online or get money from a bank I'm not looking for immersion.

What I'm looking for is a fast load and snappy ui.

When I watch a movie or play a game, that's when immersion matters.

> That video is from 2007, and yet the technology being shown is so impressive that it feels like it could have been made yesterday.

Really? Who thinks this in 2023? I thought it was a fun little demo back then, but that was it. Both then and now I just want AR as the default for everything and for screens to die already. We've had the tech for it since then too, but not enough demand from general consumers.

AR as in overlay on what you're looking at, or VR as in replace everything you're looking at?

I didn't even mention VR and I am so beyond tired of it being brought up every time AR is.

I said AR because I meant AR.

It would take some serious overlay resolution to replace a monitor, which just doesn't exist right now (?). If the angular size of a pixel is too large, it's essentially like being nearsighted. A person with 20/20 vision can see detail down to the eye forming an angle of 5 arc minute (1 arc minute = 1/60 of a degree).

Reply on news.ycombinator.com