Hacker News Reader

Comment by elromulous | Hacker News Reader

In general I agree that the tensor hardware has been mostly a disappointment. I mean this more in general performance than in ML.

In ML I think the story is a little different. Hardware design/production pipelines are measured in years. The LLM "revolution" is relatively recent. For the Pixel8, I imagine some folks at Google had to make the really tough decision of whether to run weaker, smaller models on device, or run the giant LLM models in the cloud, giving much better results, albeit slower (and with ofc the requirement for an Internet connection).

View on HN

99% of smartphone consumers don't care about 'ML performance' as a primary concern. People care how fast apps open, and battery life. From the perspective, the tensor chips are pigs that nobody is excited about.

I very much do care if my phone understands my questions accurately when I ask them, and doesn't return irrelevant results, though. Nobody says "yeah Siri is crap, but have you seen that power usage?!".

Honestly, I haven't met many people who speak to their phones. [edit: where I live. Can't speak for the north American/Anglophone market]

They usually use their phones to speak to other people, hence why signal reception and battery life is one of the most important things I look for.

All ML gimmicks are nice to show off at events, pub gatherings and in reviews videos("hey look at what my new phone can do, yeah it looks the same as the one from last year and the year before, but can your phone do THIS? <insert ML gimmick of swapping people's heads in the picture>"), but won't help me if the signal reception quality sucks and drains my battery.

Phones need to work good as phones first and foremost. High-end compute abilities come second.

My most used ML gimmick is ChatGPT which doesn't run on device anyway, nor is it something I need in case of emergency like you know... being able to call someone to come pick me up/help pe when I need it the most.

So IMHO bad modems like the ones in Pixel phones, are inexcusable in a phone no matter how fancy the other features are.

_They usually use their phones to speak to other people_

Counterpoint – I haven't met many people who use their phones to speak to other people. YMMV.

Could be a generational and culture thing as well, being Latino in my 30's and living in southern Europe regular voice calls to family and friends really are not uncommon, but I can imagine it's not common among the North American/Anglophone demographic that's a majority on this site.

But texting is also a form of speaking to people although not using your mouth.

Voice calls, the worst part of a phone. Out of anything pub-sub it really has to be the worst medium in many aspects. Long gone are the polyphonic ringtones that tried to inspire answering, now I only see vibrating or blinking phones.

With all due respect, I think you're wildly out of touch with how most people use "phones" these days. Actual calls probably represent about 5% of my phone use. The term "phone" is essentially a historical misnomer at this point.

Maybe being in my mid 30's I'm too old to be up to date on this but what do people who aren't "out of touch" use their phones nowadays for? Some form of 'Apps' I'm guessing.

And for those apps to serve their purpose (excepting the likes of Candy Crush and Genshin Impact) what do those apps need? Internet connectivity right? Which is received via the same modem as the calls, right?

So a sucky modem will negatively impact battery life and app usage in challenging environments just as much or even more as call capabilities, right?

Am I close?

Oh you are spot on.

The confusion was probably just about saying it's a phone first. I'm in my mid 30s as well and the only time I call someone is when I contact my mother. I think with the younger generation it's at a point where calling someone is seen as intrusive.

If you are calling me you are taking my choice away on when to deal with whatever you've got. Or at least are interrupting me with the attempt itself. So it better be important.

So on point. And even when it actually phones 90% of the time I just dodge it. That 10% left is pagerduty. All real contacts use some form of a messaging app instead

Can't concur at all

Won't work where I live now in south-centeal Europe (Austria ) where everything is archaic and most services are still done via phone calls like in the past: appointments for plumbers, doctors, recruiter call-backs to job applications, etc.

Almost none of those services here use email or apps as the default, everyone seems to prefer sync communication so they first tries to reach you on phone. If you don't answer or call back they'll assume you're ghosting them and don't need the appointment/service anymore and you'll miss out on important issues. Sometimes they'll leave a voicemail telling you to call them back when you can, so you can never really escape calls here without emigrating.

You can say we both live in completely different bubbles but the existence of one does not invalidate the other.

I'm from Poland. It's just over the course of years most of the services became messenger contacts, for various reasons, but most common one was the need to reliably exchange pictures

And you do need a reliable modem to reliability exchange pictures.

Yeah, and most of the time I do it on wifi. Still, most of the actual voice GSM part of my phones communication is used by salespeople

Sure but when you're out and about and don't have WiFi you'll be relying on your modem to send/receive pictures and all kinds of data.

Probably not an issue if you're all the time in cities with great coverage but you'll feel it when you go in "the woods" or well shielded buildings or basements.

Yeah, in that case you're right.

Love Austria btw., and appreciate the preciseness of your language. It's been years since I worked with it, but it always felt like a vacation for my brain, while consuming technical content in it

>Love Austria btw., and appreciate the preciseness of your language.

I'm not Austrian, I'm an immigrant here, and it's not "my language" nor anyone else's I presume, it's still just German, albeit with an Arnold Schwarzenegger accent on top.

I think people don't speak at their phones simply because the experience is so horrible.

I would LOVE to be able to voice control the various aspects of my phone but it's so mind numbingly frustrating that I'm annoyed just thinking about it. If it was better I would sacrifice other features for it.

Totally agree, the second I have to switch from a normal voice to a stupid voice with the correct nuances for the unit to "work" is usually the last time I use it.

Our google home basically sits there all day waiting for my wife to say "play abba", she has a strong non-English accent and it's about the only thing it seems to pick up. Don't get me on cars where I have to say the exact sentence to do anything.

If I want to copy text from an app (often not allowed) I now take a screenshot and copy the ocr text from there. Same with badly coded websites etc. It saves a lot of time, and is always the same workflow.

Do you see what's wrong with this picture?

That we're accepting apps so bad that we normalized the workflow of the only way to extract text from them is screenshots and OCR.

Can you imagine having to deal with this shit on PCs as well?

I agree, but it solves the problem. You are talking about how it should be. I am talking about why these local AI features are useful to me.

On PCs we have Windows with candy crush ads, so I can indeed imagine.

Sure, but OCR doesn't need ML and high-end SoCs to work well.

We had great OCR in $50 printer-scanner combo devices for home users and those didn't have M1 chips but some dirt cheap HW and Windows software to do the OCR(most likely using the Tesseract library which runs on a toaster).

So the everyday need for ML in everything in our lives is greatly exaggerated.

I am answering to this quote of yours

> All ML gimmicks are nice to show off at events, pub gatherings and in reviews videos

with a specific example of how it is useful to me outside of the cases you mention. I am merely sharing my experience.

P.S. I wish Tesseract worked as well as you mention today on my >50$ pc. I am not an Apple fanboy but OCR for images is flawless.

It doesn't have to be Tesseract specifically, there are other OCR libraries that work better and are licensed by several products(I think Abby licenses theirs).

But that wasn't my main point. Similar to Tesla popularizing ML-driven self parking, which doesn't work any better than the PID/Kalman filter methods already in use by others for the past 10+ years, my point was that today we're seemingly shoehorning ML into solving problems that have already been solved by "boring" traditional engineering solutions for over 10 years now, just to feed the AI product hype, similar to the digital camera megapixel race, or other such nonsense that offered no measurable upgrade in quality but looked good on product stickers to move merchandise at higher prices.

Unless they can do something that was going to have to be run on the CPU - and use less power doing it. I think we're rapidly getting to the point where this AI stuff is no longer optional.

iPhones have all sorts of little niceties that are directly enabled by good ML performance. For example, automatic OCR of text within images, even on websites. I think the facial recognition for grouping photos by person is also done locally by the ML chip.

Everything in photos ML related is done locally. I guess that is one of the reasons results are not great for now.

Yes OCR is better on GPT4V but my iPhone does it faster and without a network call. The difference is not that great at least in this use case

OCR is fine but photo tagging in photo app is very bad

They even recognize objects within pictures so you can search pictures via their contents. It seems to work offline, though it can take a while for the phone to run it.

This feature works surprisingly well- for example, at some point my IPhone recently gained the ability to recognise individual pets by name. This was super exciting for my kid!

the features you mention exclusively run when the device is not in use (night time, on mains) so performance should be secondary concern

Automatic OCR works fine on my Android phone through a custom ROM with the Pixel software. No internet connection or ML hardware needed; OCR isn't all that demanding.

I'm not sure about the facial recognition features, I don't use that stuff but I imagine Android phones mostly do that in the cloud. If Apple's ML chip helps there then that's good for them, but I can't say I'd buy an iPhone for a gimmick like that.

> I'm not sure about the facial recognition features, I don't use that stuff but I imagine Android phones mostly do that in the cloud. If Apple's ML chip helps there then that's good for them, but I can't say I'd buy an iPhone for a gimmick like that.

I wouldn't call it a "gimmick", it's a feature that a lot of mainstream users depend on to find pictures of loved ones. My parents use it all the time to rummage through photos and find pictures of my niece, or an old picture of my dead grandparents.

"depend" is too strong a word, given that you can accomplish the same task (finding pictures) manually as well. Also given the typical accuracy of these systems, you'll find many more pictures if you do it manually. On some photos of my dad's 60th birthday, there are 7+ people in frame and the fancy AI recognizes absolutely nobody - including my dad having a conversation front and center, who was only slightly turning his head away from the camera.

Humans use a lot more than just facial recongition, meanwhile this AI stuff seems overly focused on it. I reckon most people could recognize Donald Trump or Angela Merkel even from behind, using clues from the surrounding, their dress, stature, and hair. I'd wager you can recognize your loved ones from any angle very reliably. Can your AI?

Honestly in my personal experience the best way to accomplish such a thing is still to order by date, because you, holding your camara/phone, move continously through time/space. Pictures of a certain person usually occur together and aren't randomly distributed. An exception is maybe your SO and others you may take a photo of on a whim.

You keep saying, "honestly" and "in my opinion" and yet you are dismissive. I can absolutely see how many people can depend on it. You seem to have no idea how useful this is for many many folks. I miss it dearly in Android.

Yes, you can find all prime numbers and digits of pi if you sit down and do the calculations on a piece of paper but you don’t do that because computers are faster than that.

For more complex tasks like locating and recognizing humans and their faces, humans can still perfectly do it but I still would rather be taking more pictures of my loved ones than rummaging through my photos every single time I’m looking for something.

As for the accuracy of those systems, it mostly depends on the model being deployed.

With virtually unlimited computational budget (cloud), it’s easy to deploy a very powerful model but you sacrifice privacy and performance in high latency environments.

With a limited computational budget (local neural processing unit) you deploy what you can cram into the chip. It’s less accurate but consistent and private.

I wish I had bookmarked this comment, but someone here was dismissive of the Apple Watch's heart tracker because, paraphrased, "I just need my fingers to take my own heart rate any time I choose."

What you're arguing is both true and irrelevant. We don't even need photographs, we could just sketch whatever is in front of us, but convenience matters.

> "depend" is too strong a word, given that you can accomplish the same task (finding pictures) manually as well

Ok, do this exercise right now (if you have anywhere between 1-10k pictures on your phone): find all the pictures from 3 family members of your choice.

Clock it, let me know how long it will take you to perform this task.

Now think about my mom in her 70s, do you think it's completely doable for her to find pictures of my dead grandmother this way?

> Honestly in my personal experience the best way to accomplish such a thing is still to order by date, because you, holding your camara/phone, move continously through time/space. Pictures of a certain person usually occur together and aren't randomly distributed. An exception is maybe your SO and others you may take a photo of on a whim.

How would ordering by date give my mother access to all pictures she might have from my grandmother?

Of course that AI will not find every single instance of a face at any angle, but it will be "good enough" to find a bunch of them pretty quickly when my mother is missing her mother and wants to see a nice picture.

That's the use case, it's not someone trying to organise their pictures with utmost precision, accuracy, and completeness, it's a common person trying to find some nice pictures of a loved one. If they don't have to spend 2 hours scrolling through an endless set of pictures it's already a pretty damn good feature.

That's a bit reductive. They also care if their apps work _well_ – if their pictures look nice, if their dictation is responsive, if their assistant understands queries, if OCR is accurate… which is all about "ML performance".

I do care about my phone doing OCR and other tasks locally without Google, and tensor chips are the reason this can be done without draining the battery too much.

You're right that 99% of smartphone consumers don't care about 'ML performance', but they do care about autocomplete being good and fast. Or photos doing object detection to create stickers for messages.

If you're able to offload these 'everyday' tasks to a lower power chip, you also extend battery life. Something they care about :)

I totally imagine that they intended to run models locally when this phone was on a whiteboard in an engineering meeting but…

(1) the models team just can’t/wont trim the models, instead focusing on bigger and better. This is probably doubly true now that the AI wars have taken over their focus

And

(2) they realize they probably need to ship these features to every non pixel phone too.

Why point 2 is a problem? They can run locally if there is a chip that can do that and offload to cloud if not, and if there is a connection fast enough to transfer the data both ways in a reasonable time. I'm thinking about photo and (especially) video processing. Translating would probably be OK even on show connections. Furthermore, IMHO shooting photos and videos in the middle of nowhere with no connection is a more common use case than translating something there.

Not disagreeing with you, but when running a product and R&D team implementing and maintaining both routes is a hard sell. From a business and resource-allocation perspective there's not enough ROI, I'd rather have those resources work on improving power management.

Also, in the end the whole additional effort would be made for a device which is not well-connected to the internet, a tough acknowledgement for a company whose whole business is based on people connected to the internet.

For Qualcomm the story is different: They want to sell the chipset to companies who then build/buy local models to include in their firmware package, just like they're used to for the past 10 years...

Because you have to maintain two code paths that are very different.

One of them gets a huge budget (pixel design) and has to rewrite major ML models. Very expensive. All for a small amount of users.

The other code path has most of your users, and can use whatever the R&D team throws together. And works for every customer, and can be easily bug-fixed if issues arise.

When you’re Google with Billions of users, your long-tail use cases are all common. You have to think about what that behavior looks like regardless.

... why do they need to ship these features to non pixel phones ? It's Google's special software sauce that make people buy pixels. Google doesn't ship advances Photos app tools on non pixel phones

Because they’re not Apple. It sounds nice to gate features to phones. But they’re trying to expand their subscription revenue to offset ads. It’s a much bigger market to target every android user.

Google ships the advanced photo products to non pixels after a delay, the pixels get a head start not exclusive access.

I think this is the point - each design team and company bet on what they believe their hardware is likely to need to perform. The performance vs power consumption trade off is really tricky when you have to guess what the software landscape will look like a few years out. Get it right, and you have a winner, get it slightly wrong and you either have 'poor battery life' in reviews, or you offload stuff which ideally would be running locally (which is the suggestion in this case).

I'm not sure I want LLMs on my phone anyway - the situations where you want a 'digital assistant' are generally ones where you've got connectivity, so offloading seems like a sensible fit for me. Of course i'm making that assumption and i'll probably look silly, but that's progress right?

Of more interest is the heavy lifting for image processing to make the phone camera not suck (it's all smoke and mirrors with phone cameras, and more inference on device helps).

I'd love to have an LLM on my phone when no connectivity. There are many situations where a personal assistant would come in handy.

In the wilds if Canada somewhere:

Hey google. Lead me through the steps to create a splint and stop the bleeding. My buddy just got attacked by a bear and is bleeding out.

What? That's not a bear? It's a racoon ? Oh...anyway.

Lead me through the steps to start a fire with wet wood and how do I prepare a bear, I mean racoon, carcass for good eating. I have some old copper wire and a magnet. I need to make a phone charger. Lead me through it. Also make sure the fire is smokeless as we don't want to get discovered and sent back to the supermax.

"I'm sorry, I can't help with that."

Apparently LLM = MacGyver.

Translating that information into an easily understood format is the job of a LLM, but it's not having the data, right? Have we not moved the goalpost a bit if we want a model that contains the majority of human knowledge, including accurate and guaranteed non-hallucinated survival information, to be stored locally on our phone?

GPT-4 by some estimates is 45GB of training data. But phones can easily store 10 times that much and more if it was really required. The bottle neck is the data processing required to make inferences on that data.

Speaking of disappointing.

What happened to the Soli radar chip in Pixels?

You ever go all in on a technology that tries to sell your users on a big bezel in exchange for being able to use their phone without touching it and then have a global emergency come around that largely removes the only reliable way to authenticate the user without them touching the phone?

I’m sure Google had to think long and hard if they wanted all those queries forwarded to their data centers to analyse (enrich their user profiles).

They could process stuff locally and also send data to the cloud.

There's a big difference. If you process stuff locally, uploading the entire unprocessed data to the cloud would spark a lot of criticism. So, you are stuck running analysis locally and uploading things like relevant ad topics.

If you run your processing through the cloud, there is no optics or bandwidth cost of keeping the entire raw data as long as you want, let humans review it, and figure out which metrics to extract from it at a later point. I would say, the latter raises more privacy concerns than the former.

Yea but then they have to process everything. Sending just the relevant information for ads means they save on cost of running compute.

Considering Azure has run out of GPUs and is renting from Oracle, it’s not crazy to imagine that they could run out of TPUs and the price of compute out weighs the slightly (?) worse ad targeting.

Reply on news.ycombinator.com