Deploying Transformers on the Apple Neural Engine
machinelearning.apple.com
machinelearning.apple.com
Some examples:
- On-device semantic image search
- Text highlighting (images and videos)
- FaceID
- Computational photography, depth mapping (sensor fusion)
- Background removal (iOS 16)
- Activity detection (Apple Watch)
Siri is still far behind vs. Alexa and Google Assistant, but for every other ML/AI application I'd argue that Apple has the smoothest experience. This should be a lesson for others building ML-powered products. You don't need to use the best, state of the art models to compete. You can compete on the overall experience.
Sharing details and publications doesn't seem to be a core part of Apple's culture, so I'm glad they're going out of their way to publish more details like this article. Their push to optimize ML inference for on-device rather than cloud is going to have the biggest impact on consumer experience. I'm fairly sure this is part of their AR/VR strategy too. Low-latency local machine learning that powers magical experiences.
The iPad handwriting recognition is pretty smooth too, so long as you don't need to write acronyms / model numbers.
Amazon and Google are since the begging “web companies”, Apple comes from the hardware side and has problems keeping up with „information services“, but the Apple owned Search Engine is probably only a few years away.
Apple has been consistent in signaling that they want to push as much AI to the device as they can, for example on-device speech recognition was a heavily promoted milestone. That approach is broadly incompatible with using one enormous cloud model to provide rich interactions.
I think the goal for Siri is to become a better and better decision tree, rather than a conversation-driven knowledge engine like Google and Amazon have. I support that, I have actual humans to talk to and like that my interactions with Siri are based on simple phrases which do something and don't bottom out in a Google search or (yuck) a pitch to buy some service.
Addendum: Siri does bottom out on a search of the web, yes, what I mean is that Siri encourages use patterns (at least for me) where this basically never happens.
* Alexa (in my kitchen) almost always plays the music I ask for on Spotify. Siri (in my car) gets it wrong about half the time — often in comical and bizarre ways.
* Siri remains unable to call one of my family contacts: every time, she gets into a loop asking for clarification between two options (when I clarify, she just asks again).
* I set up an automation with Siri to start the radio playing on BBC Sounds (to stop me falling back to sleep when my alarm goes in the morning). This was stupidly fiddly to set up, and it just doesn't work: Siri tells me to unlock the phone first, which destroys the whole point.
I guess the caveat is that my experience with Siri is a bit limited, because in the light of her uselessness I often don't even try to get her to do things.
Asking Siri to turn on the lights in the kitchen: “Here is what I found in the web for ‘Turn on the lights in the kitchen’”
Asking a second time: “Ok, all the lights in the kitchen are on.”
Considering how many years it's been around, these sorts of inexplicable failures are really inexcusable. Sure, I understand nobody tests their software anymore, but "predictable" is not the term I would use to describe Siri.
Does Apple? If I upload photos from my computer it does analysis in the cloud, doesn't it?
- the image is never analyzed
- the image is sync'd to device and then analyzed
Google Lens, which isn't about sharing, does run many functions on device: for example, it does ondevice offline translation: https://9to5google.com/2021/01/25/google-lens-translate-offl...
The general direction at Google (https://ai.googleblog.com/2021/11/improved-on-device-ml-on-p...) is to move as much as possible on device for privacy and latency reasons.
There are published and opensource (Apache license, according to the github LICENSE file) models for Tensorflow Lite which runs on Pixel Edge TPUs (and most embedded devices):
https://tfhub.dev/s?q=edgetpu https://github.com/tensorflow/models/tree/master/official/pr...
But one of the things I was most impressed about Siri on the HomePod is it seemed to take context (not sure if that is the right word in this case...) into account more often.
For example:
On Alexa regardless of if I asked "how cold is it", "what is the temperature", "is it raining", "what is the weather"... etc etc etc. It would always say the same long weather response.
On my HomePod if I ask "How cold is it" I get "I don't find that particularly cold, it is cloudy and 72 degrees".
If I ask "what is the temperature" I get "its 72 degrees right now".
If I ask "is it raining" "no, I don't think its raining right now".
Then finally if I ask what the weather is I get the full thing.
Does Siri lack a lot of smarts to ask random questions and give the answer? yes, But I found when I switched to HomePod the things I normally do (like weather, timers, and similar) Siri did better.
Like I said, maybe Alexa has gotten better about this. But that is my experience. So for me Siri was better for my use cases. Home commands could use a lot of work, but that seems like an issue that no one has figured out how to make it actually better than flicking a light switch (outside of remote stuff).
I bought a HomePod to see if it could do everything I need and oddly enough, the Echo does a better job integrating with my iOS-based grocery list (AnyList). With the Echo, I can say "Alexa, add eggs" and it will either add eggs to AnyList or tell me if eggs are already on my list. With Siri I have to say something like "Hey Siri, using AnyList, add eggs to my grocery list".
I use the built-in Lists app, so I just say, "Hey, Siri, add ginger ale to my groceries list." Hopefully alternate defaults will come, the way iOS is slowly embracing alternate defaults for other functions.
That said, about 50% of the time when I say, "Hey, Siri, add sandwich bread to my groceries list," she responds, "OK, I've added those two things to your list."
Do you mean the reminders app or another one?
I think in an old version of iPhone OS or OSX it was called "Lists." That must be what I remembered.
A -5° Centigrade winter daytime temp might be completely normal and expected in my area.
However, Siri will love to tell you, like a long-lost distant relative from tropical climes, about how terrible and how uncomfortable it is.
Maybe it's regional.
Siri used to do that to me, too, which was annoying. But she stopped offering opinions on the weather about two years ago.
Disclaimer- I have never worked on Siri and don’t know. But this smacks of “hard coding” the common cases, something that apple is much more likely to do than google or some of the other big tech companies because of how poorly it scales. For me this is not evidence at all that Apple has caught up in the ML space, just that they have good enough speech to text and better UX people willing to invest in identifying and manually outlining a lot of cases.
Maybe from a US lens this is true but FB and Google (android) have much larger worldwide impact on most of these fronts where they did the leg work, published their research (which certainly aided apple who rarely publishes) and had the features out earlier.
Siri on HomePods are a disaster. You have to shout at it even for it to pickup what you are saying. Mostly when the volume is a little high.
I haven't tried Alexa and Google Assistant but from what I saw other friends operating it seems more of a smooth experience but I could be wrong.
I have the opposite experience on the HomePod mini, it's actually one of the thing that they are better at. My google home device is entirely useless to try to talk to if something is already playing on it. Compared to Siri that will understand me.
The issue is more with Siri itself being dumb and not that great.
I always speak in a normal (or slightly quieter) volume and never have an issue with Siri activating.
My biggest issue is that my HomePods need to get restarted or power cycled periodically (e.g. after a power outage or network update). I wish they were a little faster and had better self-healing but generally work pretty well as room speakers which is primarily what I got them for.
Me: Alexa, Badezimmer 10%
Alexa: Ich kann nicht Prozent auf Spotify finden
*Five attempts later, it actually works*
(It was even less reliable in English)
Siri mostly hears the right words when I ask questions, and the voice keyboard (is that the same speech-to-text engine?) is correct somewhat more often than the swipe keyboard (although that doesn’t say very much as the latter seems to have become very confused as a result of me learning German).
For fun, I set Siri to English for a day and I was blown away at how much better it is. Both in recognizing requests, despite my accent, and in the quality of the voice synthesizer. Song requests became trivial and everything felt much more fluent. English Siri is even able to express emotions (happiness), which is something I’ve never heard in the German version.
“Not you, I want the Siri on my phone” or “switch to my phone/iPad”
“Skip the song on my iphone”.
Much worse is the responsiveness.
“I don’t understand”
“Still working on it”
“I can’t seem to be able to connect to the internet at the moment”
“I’m sorry. Can you please repeat?”
With all the on-device ML stuff I assumed they’ll get Siri to operate without the internet for basic recognition and switch to network mode for more complex use cases.
- Sleep detection sort of works for going to bed, but doesn't recognize a lie in after sleepily hitting the button and dozing off for another hour. It also doesn't recognize being awake in the night, unless I actively get up.
- It detects maybe half of the walks I go on, and usually well after I've already triggered cyclemeter. (So, there's a simple heuristic that I'm on a walk -- I've told it so)
- It interprets the cat headbutting my wrist as a cue to change the watch face.
That's almost worse to me than the e-waste.
The point is about ML chops, not processing locality.
I downloaded a sample code project from WWDC 2019: https://developer.apple.com/documentation/metalperformancesh...
It didn't build on latest Xcode on the Mac Studio. I'm experienced with Cocoa and Apple's APIs, but couldn't fix the problem in 30 minutes of poking around.
Then I found another sample code project from WWDC 2020 which is apparently using a similarly named but different API for the same purpose: https://developer.apple.com/documentation/metalperformancesh...
This one looked promising, but failed with a runtime assertion and I was unable to figure it out.
At this point I wish Apple spent more effort on making their existing frameworks usable. If the only available sample code doesn't even work on a brand new Mac, the API isn't going to be used by third parties.
This is sadly not a unique situation for Apple's more specialized public frameworks — they're more like semi-private because nobody outside Apple can make them work.
Anyway, it seems that we are at dawn of deploying mode cool models which formerly required cloud computation to the hands of the users. Really cool!
Are we going to see more federated learning being pushed to user devices or is it a dead branch only useful for a few use cases?
Architectural model search is not something new. Apple has the unique access to their chip and a large enough customer base to make it feasible.
Any abstraction layer will add overhead and probably present a lowest common denominator interface. The audience willing to accept those costs is small.
Eh ... lots of people use Tensorflow or Torch.
so we add onnx and trt on top! that’ll help, right?
But if you’ve ever had to deploy these models, you’ll know it’s never as simple as promised. Different libraries use different ops, have weird export requirements or even noticeable performance issues.