Speech to Text on iPhone vs. Pixel
twitter.com
twitter.com
> Especially the offline part is very appealing to me, as it should to any privacy conscious mind. Unfortunately this speech recognizer is only available to Pixel owners at this time. Since GBoard uses TensorFlow Lite, and the blog post is also mentioning the use of this library, I was wondering if I could get my hands on the model, and import it in my own projects, maybe even using LWTNN.
Recent (May 2020) news suggests that these models may be coming to Chromium, which would make them widely accessible for offline transcription and dictation, e.g. WebRTC or video captions, https://hackaday.io/project/164399-android-offline-speech-re...
> Google is building speech recognition into Chromium, to bring a feature called Live Caption to the browser. To transcribe videos playing in the browser a new API is slowly being introduced: SODA ... What is especially interesting is that it seems it will be using the same language packs and RNNT models as the Recorder and GBoard apks
In Aug 2019, the Live Transcribe engine was open-sourced, https://github.com/google/live-transcribe-speech-engine & https://opensource.googleblog.com/2019/08/bringing-live-tran...
> Extensible to offline models
Hopefully a future version can work without the cloud API.
Purism laptops do as well, I believe.
The microphone disconnect security feature aims to make it far more difficult for hackers to use malware or a malicious app to eavesdrop on a device’s surroundings.
The feature was first introduced to Macs by way of Apple’s T2 security chip last year. The security chip ensured that the microphone was physically disconnected from the device when the user shuts their MacBook lid.
https://techcrunch.com/2020/04/03/apple-hardware-microphone-...
Having to literally lift the lid and MacOS readjusting the display every time I want to have a meeting was frustrating. It got to the point where I just bought a cheap USB microphone just to be done with it.
I support having a hardware switch for it so that I can have the choice.
I mean, the microphone is under the lid anyway so you are not doing whoever you are talking to any favours by using it in clamshell mode anyway.
I'm sure it's not much different than the test shown in the twitter post, but I've enjoyed calling myself from another phone and then screening the call and doing my best micro machines guy fast-talking impression just to see how well it really can transcribe conversation at real-time speeds.
Is it worth the hassle to try and stand a reasonable chance of failure, and then also have to do it manually, or should I just skip to the chase and do it myself.
Unless I’m setting a timer, probably just do it myself.
You can still Hey Siri for voice.
My Pixel is sat here charging. Billie Eilish is singing in the background, by the time she sings "I should have known..." it has recognised the music and passively displayed "No Time To Die by Billie Eilish" on the lock screen.
Google's team were building an ML model for music recognition and they realised oh - the smallest model fits on a phone. We don't need to spin this up as a cloud service we can just deliver it to the phones of people who want it as a feature. "Now your phone can tell you what that music is".
Of course down the road this awareness of context allows even cleverer agent interaction. "Which Bond was that?" you ask as Adele sings "Skyfall". "In the movie Skyfall, James Bond was played by Daniel Craig".
Will it contact servers to sends statistics and samples when back online?
Unfortunately, airplane mode is the only way to enable off-line dictation. Or, is there another way?
The first version of dictation had no online mode at all; when Apple added that, presumably as a Slowloris attack on the entire world, it ruined the entire experience.
Apple's online, presumably neural network based speech transcription service is horrible. It is an absolute disaster. Slow, unresponsive, drops syllables randomly, and provides no good feedback to the user. It is hard to believe that it was designed by someone who actually intended this product to be consumed by real users. It fails even basic comparisons against the predecessor that it was intended to replace.
I (accidentally) dropped my desktop Linux system into pure console mode the other day and realized how much FASTER it felt just because of the improvement in keyboard latency.
There are so many nice software touches on them that you just have to experience to appreciate.
These are seperate statements.
You can not like pixel but still consider them better than samsung.
Personally I've had one samsung phone and it nearly put me off android entirely- had I not previously had a Oneplus1 I would have considered the entire ecosystem to be pretty garbage.
Samsung is a very low bar for software/ecosystem.
Which is a shame, because I do love how unencumbered with bullshit features and spamware the phones are.
Considering the audio stream upload & processing & network jitter/lag, the speed at which text results come back is simply incredible. I don't remember the exact timing, but it was a roundtrip of 40-100?ms which is... crazy/magic.
I made a small experiment with this same Chrome text to speech engine which triggered a google image search and showed in near real time image results for the spoken words. The slowest part of that Rube Goldberg was the google images search + loading the images. [1]
I also.... "secretly" believe that very fast speech recognition is one of "the" secrets to building a smarter / better digital voice assistant. It's one of the key components (with a ton of groundbreaking NLP and/or the right regexes) that might allow closing the "strange feeling" you get when talking to the robot... voice... and... it... answers....... not ... exactly when...... you were expecting it.... to.
I'm super mega busy these days but also super mega interested in this. Reach out? :-)
[0] https://chrome.google.com/webstore/detail/dictation-for-gmai...
But what about transcription accuracy? I mainly use Android but also use an iPhone.... I have found transcription accuracy is so much higher on my iPhone then Android.
I pulled out my old BlackBerry (Android) when I sent my Android phone in for repair recently. The voice transcription via blackberry's keyboard is hands down better then either stock Google or iPhone. It's surprisingly feels like a regression going back to the new phone (other than speed and battery life)
Is this a feature only of Pixel phones?
It's not unique to the Pixel, it's just that it's a faster phone than my 5t.
But even though it's just a little slower than the Pixel, it makes a big difference. The closer it is to real-time, it seems to get so much better.
This is what Google has done and it was a monumental human achievement, deserving a place in history alongside Gutenberg, due to the small size of the model that could operate fully offline on power and space constrained mobile phones.
Compare this approach (partial mass surveillance generating privacy-preserving offline models) with the approaches of competitors like Alexa & Siri: both use mass surveillance for model training, but neither of them make their models available for offline and privacy-preserving public use.
How hard, on a scale of 1 to impossible, how hard would it be for Apple, or someone else to just grab the model google is using from the pixel and reverse engineer it and steal all their years of research.
Or trained their own architecture with data sampled from the Google model?
is it "stealing" data, the architecture, the parameters, or the act of reverse engineering and productizing the knowledge?
Models are also usually black boxes, and the techniques used are published.
If it's the former this is less interesting except for the observation that speed of conversion really is valuable.
It's a real issue for me as "4:00" has three significant digits, it's specific, whereas "4" might mean anything between 3:50 and 4:10.
1. Apple does it online, Google does it offline, hence Apple being slower is okay. But why is Apple's transcription more inaccurate then?
2. Google violates privacy because it used 411 data to train its models, hence its speech transcription quality is better. But Apple is the one doing it online, are we sure it's not using the voice data too, similar to how it has/was doing it with Siri?
It will sound like I am trying to defend/promote Google here, but that's very far from my intent.
As an iPhone user since the first iPhone, I just want iPhones to be better. I use iPhones for the same privacy concerns as many of you, but let's not give a pass to Apple. Let’s demand higher quality from them so that they spend the time, money and effort to improve. I don’t want to feel like I am compromising on a worse overall experience in favor of better privacy when i buy my next phone.
PS: Either way, I agree with most of your comment. :)
So, if the choice of secure phone is between stock Android and stock iPhone, then I think iPhone is the lesser of two privacy evils.
Stock Android devices from nearly any vendor do not suffer from these problems, and reputable vendors do even better (like in this case, where even voice transcription does not send data off device).
The thing people usually mean by privacy is “security of personal data and metadata”—i.e. being able to use your phone to break the (perhaps unjust) law, without a state actor being able to then prove you broke the law by forensically analyzing your phone.
Phones already leak a lot of circumstantial forensic evidence just by being phones. They talk to cell towers, for instance. So there’s a certain level of information leakage you’re accepting by doing something private on a phone in the first place.
The point of choosing one phone over another, for its privacy, should be to secure the phone in all the other ways—to prevent any information from leaking that can be prevented from leaking while retaining the functionality of the phone.
In that regard, iOS is usually considered the winner.
(Also, iOS is frequently considered the winner just by the fact that Apple devices can’t be interfered with by OEMs at the behest of state actors; in est, the OEM is always Apple, and so the only applicable state actor is the US. If I’m e.g. a Canadian diplomat in China and my phone breaks, I’m not going to trust a Chinese-OEM Android phone, but I might be able to trust a phone I send a plainclothes gofer to buy me from a Chinese Apple Store.)
That, too, is based in myth. More iPhones have had malware than Android phones available for purchase in Europe and the US by an order of magnitude, and Android vulnerabilities are more expensive than iOS vulnerabilities, so if your standard of privacy is protection from state actors, you should prefer Android devices.
> If I’m e.g. a Canadian diplomat in China and my phone breaks, I’m not going to trust a Chinese-OEM Android phone, but I might be able to trust a phone I send a plainclothes gofer to buy me from a Chinese Apple Store.)
If you're in China, you're in trouble because the CCP has access to all your iCloud data and any iMessage messages you send while there. Nobody is suggesting that you buy a Chinese OEM phone with who knows what modifications. Just get a Blackberry, Nokia, Google, or Android One device, and you'll be in a much better privacy situation than if you got an iPhone.
I'm assuming here that you're taking "standard precautions":
1. You get new phones from a trustworthy source, e.g. the official storefront of the relevant company.
2. You buy in person, so that the phone can't be intercepted in transit because some watchlist redirected it based on who you are.
3. You get someone likely to not be on any such watchlist to buy your phone for you (i.e. you hire some kid off the street to go into the store for you, and hand them a wad of cash to pay with.) This is to ensure that, for as long as possible, the phone's MAC address doesn't end up automatically associated with your activities. (It will eventually; but that's why you burn your phones pretty often.)
Under such rules, you won't get any "bonus with purchase" rootkits on the device. The device will only have a rootkit if all such devices in the current market have rootkits.
> Just get a Blackberry, Nokia, Google, or Android One device
You actually cannot buy Android phones in China—even from these brands—which haven't been passed through the hands of a Chinese distributor at the "OEM customization, root-of-trust-not-yet-signed" stage. (Heck, you can't even buy a Nintendo Switch in China without it going through the hands of Tencent.) Every one of these phones has a "Chinese edition" with different firmware, and that edition is the only one available for sale in China.
(How does it work for Android One phones? IIRC, the phones with these editions hit a different Android One firmware-update server, one run by the Chinese government. They still get "stock" Android firmware... in the sense that the only changes are a potential rootkit.)
The reason iPhones are trustable in that situation is that Apple has constrained their infrastructure such that they only have one firmware. That means that any China-specific customizations have to be built into that single global firmware, and activated by software (i.e. by choosing your "Region" in the phone setup.) And that means that your own diplomatic home office can inspect all such customizations using the full resources of your own state counterintelligence apparatus, and then give you the go-ahead (or not) for using the phone with such customizations.
The Chinese Android phone firmware is only distributed within China, so it's much harder to be sure you capture every version of it for security-analysis. And, even if you do, it may not tell you much about what they can do to specific people, as it may just contain "generalized backdoors" (e.g. cellular-carrier-triggered automatic firmware-update push) such that any code that actually spies on people is only pushed to the devices of People of Interest (likely with additional logic to delete itself if the phone leaves the country), such that it's nearly impossible to exfiltrate the device from China back to your own counterintelligence.
-----
But the larger consideration, if we're talking about e.g. populist recruitment into civic-action groups, is that none of those Western Android phone brands are popular in China, compared to Chinese-owned brands. Both because the state controls the advertising (so the Chinese brands get product placement on Chinese TV, and the Western brands do not); and because the Chinese phones are just plain cheaper for the same level of features (which mostly is down to vertical integration and simplified logistics with component manufacturers.) So, sure, you can try to buy one; but it'll be hard to find anyone carrying one. And you can't just rely on a random stranger to have one. Heck, just buying a phone from a Western phone-brand probably puts you on a watch-list.
...other than Apple, because Apple is unavoidably still seen as a fashionable brand in China, despite the Chinese's government's best efforts to quash this sentiment.
Under the standard of "standard precautions," iOS has had multiple orders of magnitude more malware infections than Android instead of just one or two due to Xcodeghost.
Also, my statement was about people outside of China, as you can see by the end of the quoted sentence, so many of your reasonable precautions don't apply.
> You actually cannot buy Android phones in China [snip]
You misread my point. My point was that if you're in China, you're already screwed, no matter which device you legally purchase in China. For the rest of us outside of China (including you in Canada and me in the US), Android devices are clearly superior for privacy as I have shown earlier.
> IIRC, the phones with these editions hit a different Android One firmware-update server, one run by the Chinese government.
Android One phones are not legally sold in China because the update server is run by Google, which has no servers in China.
var req: SFSpeechRecognitionRequest;
if #available(iOS 13, *) {
req.requiresOnDeviceRecognition = true
}
Pixel is doing this completely offline.
1. Google 411 was introduced 13 years ago, in 2007, and look at the ML papers published and techniques Google pioneered since then.
Is this really calling Apple’s transcription service? And does it maybe even add some latency?
Although to me, neither example is useful without:
1. Automatic punctuation
2. Robust recognition in noisy environments
And neither system is capable of that yet, although Google's system is better at #2.
Google's gotten an advantage in many fields by simply trampling over user's rights. Apple has been more respectful of user rights, and that's kind of made life harder for them too.
I'm okay with Google trampling on my rights, in exchange for the things it gives. Personalized search results, better ad hoc translation, all of these things work well only because they have my data. And for me, I like these things enough that I'm perfectly okay with the trade.
The controversy is that you don't get to choose. It is extremely hard to avoid giving your data to Google if you don't want to make that trade. You can't use an Android phone, you can't use ubiquitous services like Google Search, Youtube, GMail, or Google Maps, you can't use any website without being sure to block Google Analytics, Adsense, reCaptcha, and whatever else they own.
It's possible, but it's very hard, and requires constant vigilance. It is not a free choice in any meaningful sense.
That's fine, you're a consenting adult renouncing your right to privacy, and that's perfectly fine.
The problem is, Google is trampling on the rights of _all_ users by default, without any consent (consent, by definition, must be an INFORMED choice).
The assistant is one place (probably only place) where Google has a better privacy story than most (including Apple). Notwithstanding the phone offline capabilities, you can tell the home devices (which use online transcription) "that was not for you" and they will delete what they heard.
Having an on-device inference chip doesn't preclude them from sending back "telemetry" to improve their models. Unless they explicitly say that they don't send back data (or there's a toggle for that), my default assumption is that they do.
I think dictation on iPhone is good enough, at least 10 times better than pre iPhone / Machine learning era. But it is also not as good as google. Partly because Google has more Data and started research way earlier, and partly because this isn't a fundamental strength of Apple.
I am pretty sure Apple is working on Offline Dictation. If Google can do it with Snapdragon on a much smaller transistor budget, there is no reason why Apple cant do it with more transistor.
Look how much better the Pixel is doing on accuracy too. It's almost flawless while the iPhone, despite the advantage of offline processing, is getting maybe 10% wrong and writing complete nonsense words in there. Who is the hell is "Poor Michael Geer"?
(for clarity, I'm an iPhone user and won't be switching because privacy, but this makes me jealous)