Three chips in and Google Tensor is on life support
notebookcheck.net
notebookcheck.net
In ML I think the story is a little different. Hardware design/production pipelines are measured in years. The LLM "revolution" is relatively recent. For the Pixel8, I imagine some folks at Google had to make the really tough decision of whether to run weaker, smaller models on device, or run the giant LLM models in the cloud, giving much better results, albeit slower (and with ofc the requirement for an Internet connection).
If you run your processing through the cloud, there is no optics or bandwidth cost of keeping the entire raw data as long as you want, let humans review it, and figure out which metrics to extract from it at a later point. I would say, the latter raises more privacy concerns than the former.
Considering Azure has run out of GPUs and is renting from Oracle, it’s not crazy to imagine that they could run out of TPUs and the price of compute out weighs the slightly (?) worse ad targeting.
(1) the models team just can’t/wont trim the models, instead focusing on bigger and better. This is probably doubly true now that the AI wars have taken over their focus
And
(2) they realize they probably need to ship these features to every non pixel phone too.
Also, in the end the whole additional effort would be made for a device which is not well-connected to the internet, a tough acknowledgement for a company whose whole business is based on people connected to the internet.
For Qualcomm the story is different: They want to sell the chipset to companies who then build/buy local models to include in their firmware package, just like they're used to for the past 10 years...
One of them gets a huge budget (pixel design) and has to rewrite major ML models. Very expensive. All for a small amount of users.
The other code path has most of your users, and can use whatever the R&D team throws together. And works for every customer, and can be easily bug-fixed if issues arise.
When you’re Google with Billions of users, your long-tail use cases are all common. You have to think about what that behavior looks like regardless.
Google ships the advanced photo products to non pixels after a delay, the pixels get a head start not exclusive access.
I'm not sure I want LLMs on my phone anyway - the situations where you want a 'digital assistant' are generally ones where you've got connectivity, so offloading seems like a sensible fit for me. Of course i'm making that assumption and i'll probably look silly, but that's progress right?
Of more interest is the heavy lifting for image processing to make the phone camera not suck (it's all smoke and mirrors with phone cameras, and more inference on device helps).
In the wilds if Canada somewhere:
Hey google. Lead me through the steps to create a splint and stop the bleeding. My buddy just got attacked by a bear and is bleeding out.
What? That's not a bear? It's a racoon ? Oh...anyway.
Lead me through the steps to start a fire with wet wood and how do I prepare a bear, I mean racoon, carcass for good eating. I have some old copper wire and a magnet. I need to make a phone charger. Lead me through it. Also make sure the fire is smokeless as we don't want to get discovered and sent back to the supermax.
They usually use their phones to speak to other people, hence why signal reception and battery life is one of the most important things I look for.
All ML gimmicks are nice to show off at events, pub gatherings and in reviews videos("hey look at what my new phone can do, yeah it looks the same as the one from last year and the year before, but can your phone do THIS? <insert ML gimmick of swapping people's heads in the picture>"), but won't help me if the signal reception quality sucks and drains my battery.
Phones need to work good as phones first and foremost. High-end compute abilities come second.
My most used ML gimmick is ChatGPT which doesn't run on device anyway, nor is it something I need in case of emergency like you know... being able to call someone to come pick me up/help pe when I need it the most.
So IMHO bad modems like the ones in Pixel phones, are inexcusable in a phone no matter how fancy the other features are.
Counterpoint – I haven't met many people who use their phones to speak to other people. YMMV.
But texting is also a form of speaking to people although not using your mouth.
And for those apps to serve their purpose (excepting the likes of Candy Crush and Genshin Impact) what do those apps need? Internet connectivity right? Which is received via the same modem as the calls, right?
So a sucky modem will negatively impact battery life and app usage in challenging environments just as much or even more as call capabilities, right?
Am I close?
The confusion was probably just about saying it's a phone first. I'm in my mid 30s as well and the only time I call someone is when I contact my mother. I think with the younger generation it's at a point where calling someone is seen as intrusive.
If you are calling me you are taking my choice away on when to deal with whatever you've got. Or at least are interrupting me with the attempt itself. So it better be important.
Won't work where I live now in south-centeal Europe (Austria ) where everything is archaic and most services are still done via phone calls like in the past: appointments for plumbers, doctors, recruiter call-backs to job applications, etc.
Almost none of those services here use email or apps as the default, everyone seems to prefer sync communication so they first tries to reach you on phone. If you don't answer or call back they'll assume you're ghosting them and don't need the appointment/service anymore and you'll miss out on important issues. Sometimes they'll leave a voicemail telling you to call them back when you can, so you can never really escape calls here without emigrating.
You can say we both live in completely different bubbles but the existence of one does not invalidate the other.
Probably not an issue if you're all the time in cities with great coverage but you'll feel it when you go in "the woods" or well shielded buildings or basements.
Love Austria btw., and appreciate the preciseness of your language. It's been years since I worked with it, but it always felt like a vacation for my brain, while consuming technical content in it
I'm not Austrian, I'm an immigrant here, and it's not "my language" nor anyone else's I presume, it's still just German, albeit with an Arnold Schwarzenegger accent on top.
I would LOVE to be able to voice control the various aspects of my phone but it's so mind numbingly frustrating that I'm annoyed just thinking about it. If it was better I would sacrifice other features for it.
Our google home basically sits there all day waiting for my wife to say "play abba", she has a strong non-English accent and it's about the only thing it seems to pick up. Don't get me on cars where I have to say the exact sentence to do anything.
That we're accepting apps so bad that we normalized the workflow of the only way to extract text from them is screenshots and OCR.
Can you imagine having to deal with this shit on PCs as well?
On PCs we have Windows with candy crush ads, so I can indeed imagine.
We had great OCR in $50 printer-scanner combo devices for home users and those didn't have M1 chips but some dirt cheap HW and Windows software to do the OCR(most likely using the Tesseract library which runs on a toaster).
So the everyday need for ML in everything in our lives is greatly exaggerated.
> All ML gimmicks are nice to show off at events, pub gatherings and in reviews videos
with a specific example of how it is useful to me outside of the cases you mention. I am merely sharing my experience.
P.S. I wish Tesseract worked as well as you mention today on my >50$ pc. I am not an Apple fanboy but OCR for images is flawless.
But that wasn't my main point. Similar to Tesla popularizing ML-driven self parking, which doesn't work any better than the PID/Kalman filter methods already in use by others for the past 10+ years, my point was that today we're seemingly shoehorning ML into solving problems that have already been solved by "boring" traditional engineering solutions for over 10 years now, just to feed the AI product hype, similar to the digital camera megapixel race, or other such nonsense that offered no measurable upgrade in quality but looked good on product stickers to move merchandise at higher prices.
I'm not sure about the facial recognition features, I don't use that stuff but I imagine Android phones mostly do that in the cloud. If Apple's ML chip helps there then that's good for them, but I can't say I'd buy an iPhone for a gimmick like that.
I wouldn't call it a "gimmick", it's a feature that a lot of mainstream users depend on to find pictures of loved ones. My parents use it all the time to rummage through photos and find pictures of my niece, or an old picture of my dead grandparents.
Humans use a lot more than just facial recongition, meanwhile this AI stuff seems overly focused on it. I reckon most people could recognize Donald Trump or Angela Merkel even from behind, using clues from the surrounding, their dress, stature, and hair. I'd wager you can recognize your loved ones from any angle very reliably. Can your AI?
Honestly in my personal experience the best way to accomplish such a thing is still to order by date, because you, holding your camara/phone, move continously through time/space. Pictures of a certain person usually occur together and aren't randomly distributed. An exception is maybe your SO and others you may take a photo of on a whim.
For more complex tasks like locating and recognizing humans and their faces, humans can still perfectly do it but I still would rather be taking more pictures of my loved ones than rummaging through my photos every single time I’m looking for something.
As for the accuracy of those systems, it mostly depends on the model being deployed.
With virtually unlimited computational budget (cloud), it’s easy to deploy a very powerful model but you sacrifice privacy and performance in high latency environments.
With a limited computational budget (local neural processing unit) you deploy what you can cram into the chip. It’s less accurate but consistent and private.
What you're arguing is both true and irrelevant. We don't even need photographs, we could just sketch whatever is in front of us, but convenience matters.
Ok, do this exercise right now (if you have anywhere between 1-10k pictures on your phone): find all the pictures from 3 family members of your choice.
Clock it, let me know how long it will take you to perform this task.
Now think about my mom in her 70s, do you think it's completely doable for her to find pictures of my dead grandmother this way?
> Honestly in my personal experience the best way to accomplish such a thing is still to order by date, because you, holding your camara/phone, move continously through time/space. Pictures of a certain person usually occur together and aren't randomly distributed. An exception is maybe your SO and others you may take a photo of on a whim.
How would ordering by date give my mother access to all pictures she might have from my grandmother?
Of course that AI will not find every single instance of a face at any angle, but it will be "good enough" to find a bunch of them pretty quickly when my mother is missing her mother and wants to see a nice picture.
That's the use case, it's not someone trying to organise their pictures with utmost precision, accuracy, and completeness, it's a common person trying to find some nice pictures of a loved one. If they don't have to spend 2 hours scrolling through an endless set of pictures it's already a pretty damn good feature.
If you're able to offload these 'everyday' tasks to a lower power chip, you also extend battery life. Something they care about :)
What happened to the Soli radar chip in Pixels?
modern AI is just vector multipication. any AI chip is just 10,000s of very simple cores which can do vector float operations and little else. this also entails clever trade offs of shared cache and internal bandwidth.
(as a thought experiment, consider a naive million by million matrix multipication. this will take a single cpu about 1 year! how do we reduce this to 1s ?)
the end
I thought they were generally int8 or int16 vector multiply adds and occasionally float16 added in.
Each "unit of work" in matrix multiplication is not dependent on any other unit of work. Stuff as many cores as you can into a chip, and then simply feed in all your vectors at the same time.
I.e. basically a beefed up GPU or an "AI" chip.
A matrix of that size in single precision is 32TB, a better question is how do you store it?
I'm not buying it. I have a couple of Google's (mini) Edge TPUs doing object recognition in my Cctv server. They have been working 24/ 4 for about 2 years and are consistently delivering on the 4W/4Tflops promise(although for some reason they stopped advertising this number focusing on benchmarks instead).
I for one wish Google would have us their big (or even medium) sized TPUs to play with on an add on card rather than in their phone.
Yeah, no kidding. I know they want to push GCS, but what a boon that would be for TPU support in ML projects.
“My car isn’t so good at driving.”
Kind of sounds like life support.
LLMs are hopelessly memory bandwidth bound. Most other models are compute bound.
Snapdragon can't breathe fire.
When someone tells you to "touch grass" that includes eudicots as well: https://en.wikipedia.org/wiki/Antirrhinum
If you're mocking that entire idea, then that also supports the claim that it's "on life support".
Maybe Google can't market anything because they're so entrenched in the least effective advertising methods out there?
Where are the catchy short commercials on YouTube compelling me to this great tech? Just because everyone knows Google doesn't mean they know what Google has to offer.
It wasn’t good enough. It purported to be great, but it had random connection issues. Given that destiny, a 1st person shooter, was their flagship game these flaws really jumped out.
You also really needed good internet. Like really good. Mine performed well sometimes when it was hardwired, but lagged regularly over WiFi.
I think these things were fixable, but that thing was dead within a month of launch. Google didn’t get fully behind it, so all the people that launched it got their promos, saw the writing on the wall, and transferred out.
In the earliest days of the internal dogfood, there was one game available: Doom (2016). This was very much intentional to make sure the streaming team focused on the twitchy latency-sensitive gaming segment.
> You also really needed good internet. Like really good.
What was more important than really good internet was stable bandwidth -- even a 4k session could work with a 25Mbps downlink, but very few providers will actually give you that guaranteed bandwidth. Similarly, having gigabit speeds from Comcast doesn't matter if you randomly see spikes of bufferbloat where your latency to the server jumps from 10ms to 100 (congrats, you've now dropped several frames).
WiFi similarly struggles to give you predictable delivery (apparently 20ms is a good number for wifi jitter, but that's pretty much the entire latency budget we had allotted for networking).
> all the people that launched it got their promos, saw the writing on the wall, and transferred out.
This is untrue. The shutdown announcement took the whole org by surprise (other than senior leadership). There were hundreds of engineers who were effectively told "you have X months to find a new position in the company, or leave".
It is true that Google didn't get fully behind it, senior leadership was worse than ineffective, marketing was terrible, and our reputation of killing off products meant it never reached the level of market adoption needed to sustain the product. But we had absolutely done our research to figure out the size of the addressable market and what levels of stable bandwidth users could sustain in practice, well before launch.
I am not sure if you'd want one? I returned the Pixel 7a after a few days. Stock Android is great, the camera is top-notch and goes head-to-head with iPhone. But the fingerprint sensor is mediocre (they use an optical sensor) and charging is very slow. The issues make it not-great as a daily phone. People have reported similar issues with the 7.
Returned the phone and got an iPhone 13 instead; and as painful as it was switching to iOS after 13 years of Android, I don't regret my decision. The iDevice comes with its own niggles, but at least I don't have to worry about battery life (yet). Maybe google will figure it out by the time the Pixel 8a comes out.
Also the author assumes Google wants to do on-device AI processing, but given Google's desire for user data they may be uploading pictures and videos in the cloud just for the sake of having the data (and maybe use it to train say self-driving AI models ).
I could live with the modem issue (3.5G is OK for 99% of my smartphone usage anyway), but having to care so much about battery running off is a quite important drawback for daily life. The fact that they seem to be keeping the same path with the chip makes me think that I'll have to look for another brand for my next phone.
Never had issues with reliability though, I've been really impressed with the hardware aside from the battery.
"The model is also quite unreliable" should read "The modem is also quite unreliable".
I only have reliability problems with the data connection (specifically 4G and 5G), in other aspects the phone works fine.
Granted I am not a heavy user of things that might need extreme capabilities. I do not play complex games or do media creation.
Tensors are not as fast or as efficient as the alternatives (the reason why I didn't pick the 8 Pro recently and went with a Galaxy S23 Ultra), but it's fast enough for most users and the efficiency could probably be fixed by using TSMC instead of Samsung, just see the difference between Qualcomm's 8Gen1 vs 8+Gen1.
Every time you use Google AI that’s offloaded to their cloud you’re basically getting free “money” from Google, and all of these functions are places where advertising can’t be inserted in a way that makes any sense.
The most profitable cloud services are things like “sell storage that sits unused” or “sell a subscription to fitness classes.” If you want to run something complex and resource intensive you better hope you have some enterprise customers lined up.
They won't ever give up their ability to have control over our data.
Hey Google! Please do this thing for me.
I'm sorry Dave, but I can't do that. You must be plugged in and connected to WiFi.
Hey Google! I'm at my kids sportsball event, and that's not really conducive.
I'm sorry Dave, I didn't get that.
Besides, we're talking about 3 chip designs and still not there for on device processing at the same levels as competitor(s). So just punt and send it to the cloud.
interesting, these are raspberry toy processors, maybe they should have released 'real' ones.
Siri is astonishingly, infuriatingly bad. It's hard to imagine anyone on the Siri team uses it daily like I do. It works well ~30% of the time and is bafflingly non-deterministic.
Google is much, much better, but it's still not objectively great. It's closer to 80% of responses being reasonable, which is still unacceptably low.
It would seem that an investor bet like with bitcoin would be on the specialized SoC not the tech above the chip itself.
I gather PyTorch on Google hardware now works but it’s kinda stolen their thunder
Fact of the matter is that google doesn't make good interesting and innovative products because it has started drinking it's own cool aid.
About at the same time as when they started becoming evil, getting political and influencing elections everywhere, they also stopped even trying to make innovative products and instead went full communist.
Everyone knows the effect that has on innovation but somehow SF techies and american wokies are unable to comprehend it, and vehemently deny it.
Would you mind elaborating on this point?
I heavily doubt that Qualcomm is even capable of working in such a role. In fact, the C-suite probably believes that their core competency is lawsuits and not modems, anyway.
[0] https://twitter.com/calebccff/status/1472517091970494465
(Edit: I meant semi-custom SoC design/architecture not cores, like https://www.anandtech.com/show/7281/understanding-amd-semi-c... )
Yes, the early generations of Snapdragon SoCs were built using a custom ARMv7 core called Scorpion, which was later succeeded by another v7 core called Krait. The first ARMv8 SoC (the infamously disastrous Snapdragon 810) used stock ARM cores, but the next generation (820) switched to an in-house v8 design called Kryo. Around that time though, the company did massive layoffs, including in silicon engineering, leading to all the subsequent designs using tweaked ARM cores (despite also being called Kryo)
It's not 1 of capability. They're trying to force everyone to use the full SoC and not just the modem.
Google doesn't have nearly the same leverage and volume numbers to use against Qualcomm as Apple does, but even for Apple it's a devil's bargain. The licensing terms are incredibly onerous. If they updated them and you didn't agree, they used to just cut off your entire company's chip supply until that was ruled monopolistic and illegal. The royalties (one of the only royalties Apple actually pays) are substantial percentages of the final device price. Getting into bed with Qualcomm needs to be weighed against the risk to the rest of your business.
As a consumer, I buy Qualcomm modem phones, but as an engineer Qualcomm is only marginally below the plague on my avoidance list.
It's just a shame, even now I find myself eyeing the black Friday sales prices. I loved everything else about the phone, but I know that the battery life is going to drive me mad and with all the heat and increased number of charge cycles there's just no way the battery is going to last as long as the support does and that's a deal breaker for me.
Then my coworker got the Pixel 8 Pro, and it has WAY worse battery life than my 7 Pro, he even went and got a new one at the store because it was so bad it had to be a fault, but no. It just is absolute garbage.
As an engineering lead I do a lot of my non-programming work directly on Pixel, including tons of remote desktop, meetings, GitHub/productivity, k8s management, etc. I don't get to write code nearly as much any more, probably around 10-20% of my work week. At least for those workloads, things are fine. But the workload you run can dramatically change things, especially if you've got particularly power hungry apps involved. The OS gives you tools to identify what those apps might be when you dig into the battery area of settings.
> even without google play services. (came from a 5a)
Since you can't remove GMS from a Pixel without flashing a ROM to it as far as I know, can I assume you are not using stock software? A lot of Pixel's battery performance is made up using the adaptive battery features. When they are off, things are more dicey. Do those work on custom non-GMS ROMs to begin with?
To sum it up, battery can definitely be better, but I'm just not seeing the kinds of problems people complain about and that surprises me given my experiences.
I wonder if there's a Termux equivalent for iPhone. It's the only thing keeping me on Android at this point.
If my financials got better, I'll definitely move to iPhone