Apple Vision predictive bio-feedback
twitter.com
twitter.com
The weird phrasing in the tweet is because I can't talk about any of the work I did due to NDA's besides my job description and those patents. But I had a pretty unusual role all things considered. I was also trying to clarify in the tweet that these aren't meant to be interpreted as unannounced existing features or capabilities of the product. But rather, research I did and contributed to, that was made public through patents.
There's a ton of interesting preexisting research literature about neurotechnology research for VR. Both in terms of using it for biofeedback, and using it as a device just for studying the brain. If that sounds interesting browse through what's out there via google scholar.
Also I said I spent 10% of my life working on it, realizing that was a goofy way of putting it. But it's a standout thing in my life and it was mostly a self reflection on what a trip being alive is, not trying to brag about it, mostly just sharing what I did. It's an unusual product and I had an unusual role
Anyway it's a cool device and hopefully people enjoy it.
Same for me! Didn't even notice any weird phrasing (the tweet got immensely popular though, so nitpickers inevitably arrived). It was also nice to know people are working on getting non-invasive neurotech into the mainstream - very exciting
How to break into it as I have many ideas Id love to see if they are any good. Are there bootcamps to quickly learn development and around other fellow other eager developers? Im a web designer who uses basic code (html/css/jQuery) to design.
Would love to chat. Can you drop me an email? $HN_username at gmail
I'm more concerned with ad tech that's already out there and having massive impacts on society.
I would only be surprised of this outcome on no -Apple posts. It's quite clear there is mass adoration (or cult following, you judge?) of Apple on HN
It's always magic, except when the predictive model doesn't work and there is no corrective action possible on the user side.
We're talking about physical reactions, so that means it's supposed to be the same through gender, age, race, health conditions from the algorithm's point of view...That's a lot of variables, and I wonder if Apple have analyzed this beyond the US population, assuming the device will be sold internationally at some point.
Apple seems really confident about this as they are foregoing controllers on the default interfaces, but that also reminds me on how they were rumored to make a car without a driving wheel...fingers crossed, I guess.
I expect people for who eye tracking fails to be stuck on third party controllers for the default interface (hopefully they're supported outside of games ?).
The bummer though will be that this device is US only for its launch, so the first round of application might also not care at all about these kind of issues. Accessibility is something Apple usually cares about, but third party devs can be more casual about it.
But instead every screams about dongles.
Quality Thunderbolt 4 hubs cost about $150-500, which is fine if I need to extend the selection of ports at my desk, but there's no point in having everything on one USB-C port just because it's technically possible.
It's what i would do as well if I would want to steer the developer ecosystem in a certain direction.
From what I heard it also supports Bluetooth Mice and Keyboards to productively use a Macbook (which is streaming its video into the glasses)
If they ever opened the system to run MacOS, that is, which is very unlikely :(
But I guess they won't, because (a) it's probably a bad idea to put unnecessary computational load on the headset that will use power and heat up your face, and (b) because they want to keep selling you Macs.
On Mice/Keyboards, I hope this supports third party VR controllers, otherwise the gaming/fitness side is really toast. That feels like a given, but considering the time it took for iOS to get Nintendo controller support, it's not unprecedented, especially given their relation with Meta.
It kinda makes sense that some of her traits would be covered inside the test pannel, but looking at the face unlock and the level of adjustments they're still doing, it doesn't look like they got nearly enough data before launch. And that's for a product that is selling in billions of units.
Also such a thing as being too conservative with battery size (iphone 6?) so that when it unavoidably degrades, you must throttle down the phone to avoid killing the battery and perhaps more (literally by a current draw it can't handle since the internal resistance has increased).
And that's just apple, then there's unintended acceleration (toyota), exploding airbags, faulty ignition (ford was it?).
Don't think that just because they are large, or a company, or old, or whatever, they always have their shit together. Companies are people. People make mistakes. Mistakes can slip through the cracks.
This is it. They've pivoted and instead of a car, you get to use Vision with a karting game
This part makes me sad about patents and original idea behind them. This actually is the opposite of original idea why people came up with the idea of patenting something. To spread the knowledge and to award inventors. Not to stifle the competition.
The point of patent protection is that you teach everyone how to build the thing that's being protected, in exchange for a monopoly.
Not actually teaching people how to build the thing that's being protected, but still having a monopoly on it is the worst of all worlds.
If you aren't going to teach us how to build something, don't expect patent protection for it. I'd rather you keep it as a trade secret, good, go do it, it's a free country, but then we can just reverse engineer it. If I don't have an NDA with you, I have no reason to keep your trade secret... secret.
Sounds like he cleared the post with Apple, though.
(tho i personally still think the patent idea is probably misguided -- apple would develop this tech ANYWAY, and it will all become public knowledge once people play w/ the devices enough, so what are we really gaining here?)
but still the argument can be made.
Whether or not those trade secrets were likely to leak before or after the expiration of the patent... well... in theory the patents mean you get the info out there even for the failed companies where leaks wouldn't be interesting. In practice you've got patent trolls.
I think it would make sense to shorten the patent length for certain things, based on how fast the state of the art in that industry is moving, but it would be hard to make everyone happy with whoever was in charge of deciding those terms.
With patents: You are provided with useless information. Information that is literally deliberately designed not to be useful. You can't practically speak use it. If you do obtain the information in any other way, you'll be sued into the ground. You'll be sued even if you've never seen the patent. You'll be sued even if your own invention is only vaguely similar. You'll just be sued, let's leave it at that.
now we all have to _never even look_, because that might support someone's claim that you were aware of their patent and chose to violate it anyways - triple damages
In which case patent regime is working exactly as intended and promoting dissemination of ideas while still providing the (time limited) economic advantages to the inventor.
Had he wanted to state that the patents were carefully written, he would have written "which", not "and".
"All of these details are publicly available in patents, and were carefully written to not leak anything."
I'll simplify:
"Details available in patents, carefully written to not leak"
What's been written? The patent.
What is in the patent? Details.
How are the details written to not leak? Carefully.
What are we reading? A tweet.
What does it contain? Details.
What details? Ones that are publicly available in patents.
How have they been written? Carefully, to not leak anything.
The details are contained in the patents. The tweet explains some of the high level concepts (but not the details - after all, it's a tweet). Neither - honestly - are written to leak anything (like trade secrets).
The tweet goes on, saying if you want more information to hire them. It's a pitch for their consultation services, since they believe this tech will mature in the coming decade and, don't ya wanna get on the base floor?
Seems pretty cut and dry.
By definition, patents don’t include trade secrets. They’re patents. If you publish something in a patent, it ceases to be a trade secret.
And of course patents are written carefully to choose what to disclose. And the things that they don’t disclose remain trade secrets.
All the author is trying to do in their statement is to make it clear that the only details they plan to include in their tweet are those which have already been disclosed via patents, and therefore none of the content could possibly be an Apple trade secret.
That’s not because they are carefully trying to conceal anything from you.
It’s because they are carefully trying not to get sued by Apple for disclosing trade secrets.
(None of this comment should be interpreted as a defense of the travesty that the patent system has become)
It’s entirely possible that the patents created for the Vision Pro are good enough to recreate many of the individual components that have been patented, but don’t help you assemble the entire product. Which I would argue is entirely within the sprit of the patent system. Just because you want a patent an idea a receive a monopoly on it reproduction doesn’t mean you need give away the details of how your entire product works. Indeed you may probably wish to keep as much stuff a possible hidden a pure trade secrets, if you’re confident that they can’t be easily reverse engineered or replicated.
But none of that diminishes the value of the knowledge shared in a patent. Nor does it inherently mean that the patents are written in a way to be entirely useless, they’re just extremely specific and don’t include broader context that isn’t needed to reproduce the specific thing being patented.
For example you could patent “estimating human emotional intent from body worn bio-sensors”, without once mentioning its intended for use in a VR headset. Just because you intentionally avoid mentioning a VR headset doesn’t make the information in the patent useless.
I don't think so. The spirit of the patent system is you submit your product to them in the form of a working model. You get to patent that device. Tearing it apart into individual inventions to me is against the spirit.
Of course none of this matters anymore, we are far down the rabbit hole.
On what basis? The patent system protects inventions not products. Not all inventions are sellable products in themselves, and not all products are inventions.
How would you patent anything meaningful if you can only patent an entire product? It’s would be trivial to around just patents, just make a minor insignificant change to the design of the product, and bam, you’ve got something brand new and unprotected. Thus rendering the entire reason for getting a patent meaningless.
This is standard CYA language from an employee who doesn't want to get in trouble for leaking anything not publicly known (e.g., something not in a patent document).
The Free Software people were right - a device this intimate needs to have publicly available, verifiable source code.
It also induces different states by way of imperceptible stimuli and measured the response.
The tweet got increasingly dystopian the more I read; spine-chilling stuff. Had this been about the Quest, people in this thread would be reasonably apoplectic. Imagine what Saudi Arabia or a Gambling or News app could do with this.
Fuck it, I'm going Thiel, anyone want to cofound a security startup, HMU (law enforcement experience advantageous) - we'll make the Apple Vision the best interrogation tool on earth. Fifth amendment? Pssh, the suspect won't need to say a single thing.
Are you sure? Am I missing that part? I see this:
>Other tricks to infer cognitive state involved quickly flashing visuals or sounds to a user in ways they may not perceive, and then measuring their reaction to it.
But that doesn't say they're inducing states but rather trying to figure out the state. Trying to understand if someone is mad does not seem quite as dystopian as trying to make someone mad.
Classical conditioning has been known since Pavlov and his experiment with a dog.
For starters (apart from a very few outliers), if you own any Apple product, you have fallen into the trap of paying more for something that you believe is more valuable to you, but in reality, that feeling is artificially created. You aren't paying extra for features, you are paying extra because you believe that those features are worth that much, which is not your own thought.
I don't really know how I feel about this...
It's true with all new tech these days, and I'd be doubly worried when it comes to things like emotion detection. There are no good uses with strong business cases, but plenty of nefarious ones.
Reality: "We'll use it to store media encryption keys, so hollywood will let us stream media to your device."
I do not even carry a smartphone anymore and am happier for it.
A totally closed hardware & software combination tethered to a corporate cloud ? NO WAY IN HELL!!
You'd also have to never talk to another person with them, since other people are capable of manipulating your emotional state and can't be audited.
> Other tricks to infer cognitive state involved quickly flashing visuals or sounds to a user in ways they may not perceive, and then measuring their reaction to it.
The post is talking about past research work and about patents; Apple has lots of unused patents.
But the headset of course does !
Otherwise, it could be interesting if as users we got access to all these measurements, for our personal use. That could open the door to many other applications that would be probably less creepy.
You can! It's not trivial, but not very hard either.
eye tracking - OS projects getting this from a laptop camera exist and work well enough for assistive tech (i.e. it can replace your mouse today)
electrical activity in the brain - there are fancy things like https://zeto-inc.com/device/ but there are basic options as well
heart beats and rhythms, muscle activity, blood pressure, skin conductance - there's lots of cheap sensors for those
Pupils dilated, heart rate racing, therefore “duck” may not be the right word..
I would not bet on it.
-A VR show can modify the jokes in real time if they know how funny you find them.
-A VR casino can offer a 'full biometrics' table for the players who really think they know how to bluff
-A VR porno can have your VR GF climax at the same time as you
This seems like an oxymoron. I get that it's a "virtual" girl friend which semantically qualifies the term girl friend, yet there's something immensely sad about the combination of these terms.
VR sex bot seems more apt to me. It doesn't really matter and I don't mean to impose terminology on anyone; it's just an observation/reaction.
The ability to form a connection and understand your needs perfectly and respond to that a companion is the real value
Sex gets boring pretty fast . It is not that expensive to pay for in real life, a companion would be supportive, knowledgeable, empathetic and always available is very hard to pay and get or have otherwise .
I don’t know why people find it sad , not everyone is able to find the perfect (or any) human partner /companion , we should be happy if we are all are able to connect to sentience does it matter if it is a human , dog or a machine ?
- they can track what you fear
- they can track what you like
- they can track what enrages you
Agencies will love that data.
As isolating as it needs to be to extract max revenue. Your fine-tuned proclivities profile will be one more thing to lock you onto the platform, and an opportunity for product placement/ ads. Marketers would love to have their brand associated with happy thoughts (and happy thoughts only).
Emotion recognition is an old field but (apart from autism research) the only application i can think of is emotion manipulation. How does one wake up in the morning and decide to work on consumer emotion recognition
I think two significant factors would be 1) they have the skills to make progress in the field and it's highly rewarded, and 2) they believe there could be real value in making progress. One or both of those could be enough to motivate a lot of people.
5000 patents filed from Apple is crazy, and looking at the thing with all of these new both pieces of hardware and concepts, wow.
How many people have been working on this, and for how long? The other threads here seems to have missed just how "huge" a projects this is.
Is this also the lead up to the Apple car, or some other ecosystem?
The other threads are busy jumping on the Hate Apple bandwagon, is what it is.
I have my own criticism of Apple's VR goggle, specifically that its form factor (albeit necessitated by today's technologies) drags the whole thing down in both the luxury/fashion and usage factors.
But if I put that aside, the technology and innovative thoughts crammed into that thing is potentially groundbreakingly amazing if the marketing is truthful and accurate. Even the $3500 price tag, which yes is bloody expensive, doesn't sound unreasonable given most other "cheaper" VR goggles still require a mothership costing about the same anyway.
I wish Apple success here, because the VR industry needs someone to come in and kickstart it back to something interesting and worthy of attention.
...I also admit I wouldn't mind seeing all the haters getting put out to dry for their noisy trouble, but maybe that's beside the point. :V
When the tech is open source and I can control what it does, then sign me up.
They've never gotten any profits from selling targeted user data. Mobile gaming marketing units are _still_ butt-hurt about Apple's randomised user/device IDs that prevent them from tracking where users came from and downloaded their game.
FB and Android tracking still works perfectly, but those users are less valuable because they don't spend money.
Enough companies have tried the cheapo approach and failed. Even mid-price ones are still niche devices.
Apple went full-on to Varjo[0]-levels of tech inside the headset. Now developers have 6-9 months to bring their A game.
Then Apple can see what kind of applications and use-cases stick and start figuring out what parts of the Vision Pro they can downgrade or leave out to make a Vision Regular, which will come out in 2-4 years along with Vision Pro 2.
XR has the potential to be the next general computing platform after the PC and the Smartphone.
This might be controversial, but I believe what the VR industry would need most is for the giants to agree that collaboration for a wide open ecosystem yields the best result for everyone.
Unfortunately we may have another company here which is not interested in such a direction at all. What we get instead is the biggest giant of them all forcing through it alone with yet-another closed ecosystem in expectation of taking full domination of the market and all its innovation. But this time it's the player with the highest chances of success.
Now imagine you're a trillion dollar company.
People and Media are not so much interested in technology research, and without a product it's also often too abstract to understand. If a company makes no publications or speeches at conferences, they can often work for YEARS without any special measureas and not getting noticed...
The product itself they then likely developed with the same business-practices to secrecy as an iPhone.
Not all employees are working on top secret projects, those who do very few get the overall picture, they just work on some individual features.
Eg https://www.zillow.com/homedetails/20488-Stevens-Creek-Blvd-... recently sold.
Anyway, I'm not accusing Cupertino residents of being upper class, I'm accusing them of being old.
Dickens himself couldn't do a better character introduction than these two sentences.
I can't say I really understand the art - and I went to school with him, had the same professors, etc. I'm just scratching my head when I read his tweets.
But well: he's worth many millions and I still have housemates so I guess capitalism has crown him in the lead.
Is anything indicating that solutions mentioned in the post would actually become part of Apple Vision?
> measurements like eye tracking, electrical activity in the brain, heart beats and rhythms, muscle activity, blood density in the brain, blood pressure, skin conductance etc.
The headset does not appear to track most of these things.
Apple’s buzzword of the day yesterday was “on-device processing” and “on-device X” in general. They have invested heavily for years now into shipping ML chips in their devices and don’t seem to be stopping; if anything they’re increasing the use of it.
Is there anything definitively claiming this would be any different?
I'm talking about the infrastructure not this device necessarily
"Where a user looks stays private while navigating Apple Vision Pro, and eye tracking information is not shared with Apple, third-party apps, or websites. Additionally, data from the camera and other sensors is processed at the system level, so individual apps do not need to see a user’s surroundings to enable spatial experiences."
sorry, but the eye gase vector will will easy to infer by where its rendering stuff, and where the gesture system is triggered.
Sure apps wont get a look in to the shape of the room persay, but again you can infer it when looking for occlusions.
I want to measure my pupil distance while browsing social media. The reasoning is my reading in "Thinking Fast and Slow" that pupils dilate when we see something we find interesting / something we like / when we are thinking and vice-versa. I want to put it to the test and it seems like finally the consumer tech is close to making it possible.
Does quest support pupil tracking? I did some cursory research but couldn't find any reference for it. Industry pupil tracking headsets are way too expensive; My last hope is that someone will jailbreak vision pro...
Is this like how in the old days the movie theater flashed a single film frame of BUY POPCORN YUM.
I wonder if/how they incorporate that.
Because a lot of social drama comes from humans betting on the statistically likely thing in social situations and it being wrong rather than checking or hedging their bet. ("I shall assume the white male is in charge. Oops. Now I've offended the woman or person of color who is actually in charge.")
Source: Thinking Fast and Slow, where the initial inspiration for the book was an experiment noting that human's pupils reflexively dilate when they engage their "system two," aka the "slow brain," metaphorical part of their mind.
Even then it only really works looking dead ahead with your eye wide open.
so for the moment, that stuff is firmly in the lab.
So whats the real play?
to replace phones with glasses that know you better than you know your self.
To bake all that you see and do into <300 megs a day, and make it searchable in a way thats useful to you (not as in where was I at 22:04 on the 4th) as in "when did I last see x, what colour y did they say they liked?" but without having to ask that question.
Imagine a really really good personal assistant, that could answer 99.5% of the boring questions asked you with authority. That is the level of AI they (and google and facebook) want
Facebook, for instance, makes money by showing you ads, not with "your data".
Apple takes an idea and staples as many 'side-grade' style functionalities to it as possible in order to sell it , rather than just offering a really well-performing product within the niche.
The iPhone was a remarkably bad phone -- it was more awkward to hold than any other phone at the time (now they're ALL awkward), and the reception was terrible. The provider it was locked to at the time was one of the worst in North America at launch; but the iPhone had features that people went nuts for which drove sales beyond the phone offering.
I feel like this VR headset will be similar.
It's a mobile device with a 2 hour battery life -- something that everyone in the world has been saying would make a device dead on arrival for the mobile market; but it has all of these little side-grade features that will attract purchasers that are interested in the Apple product itself but not necessarily VR.
As a consumer i'm glad -- I may have never been a big iPhone fan, but the Android reactionary effort produced so much value that I can hope that this field will see something similar once it proves the market feasibility. I can't wait for my 8 hour/8k res/quarter-priced white-label VR headset clone in a few years.