Hacker News Reader

Comment by summerlight | Hacker News Reader

I'm surprised that no media outlet is seriously talking about its on-device inference capability, which is the biggest next thing for LLM. Apple did a better job on its PR here, but they didn't also get all the attention that it deserves.

On-device inference improves not just its latency, this also removes a huge chunk of LLM's economical constraints from software companies. The biggest advantage of software is its marginal cost being nearly zero. LLM hasn't enjoyed this luxuries but the dynamic is going to change.

View on HN

People are getting pretty tired of AI. Just about every tech company is shilling "AI", without actually selling anything which is useful for most consumers. On the other hand, just about everyone is being drowned in AI-generated garbage. To a lot of people, "on-device inference capability" is identical to "FlarzBuz engine to power Garbage Generation".

Also, I'd disagree with it going to change the economics behind it. Sure, offloading the hardware and power cost of inference on the consumer is going to make it cheaper for the AI companies, but they're still going to have to spend an absolute fortune on training. And how are they going to sell it if it runs on your device? The tech industry has been trying to push everything away from on-device code in order to sell subscription-based cloud services! Who's going to pay a subscription or per-query fee simply for the right to run a pretty mediocre app?

Also promising AI isn’t a new thing for Google.

They’ve over promised and under delivered on AI for years, and that is now their reputation.

Going back and watching the Google Duplex demo versus what shipped and what it is today - that was 2018 and it still doesn’t perform as demonstrated.

After the media backlash, the team was quietly deprioritized.

Source: knew folks on the team.

Easy, they just disable it if you don’t pay. Pretty sure automotive companies are going to make breaks a paid upgrade next.

For years, Google has been promising me a high-performance voice assistant. I find that in 2024 it still can't understand "call mom" and "call my wife's name" unless I'm in a perfectly silent environment. And it usually takes several tries.

I'm not going to get carried away with technical announcements.

Telling my Pixel 8's assistant to "set an alarm for 10 minutes" fails when it's not connected to the internet (after spinning for a few seconds.) That's crazy to me. I thought offline voice for basic things was working back in the Nexus days.

I travel a lot and often find myself in areas with poor or no connectivity, and things "get a little freaky" in those scenarios. Even things like Google Maps (which has downloaded/offline mode) often fail or spin for 30 seconds. I think Google/Android engineers don't test these scenarios enough (or at all), and they're usually bathed in perfect connections.

Siri seems to be better at the voice recognition in my experience but it has these crazy bugs where I see it perfectly transcribe the text, and then un-correct it into something completely different. Piping the raw text into a 90s Perl script would be better, and it amazes me that the same team can do on-device transcription so much better.

I dont want to use the voice assistant on my pixel but the function is so baked in that if I connect a headset, despite never setting the assistant up, it will interupt any music or playback after 15 seconds to tell me the assistant is ready now and there is no way to make that message go away or resume my media playing.

What's even worse for me is that it regularly "forgets" how to respond to queries that it could serve perfectly well before, especially related to smart home things.

"Play <local radio station x> on living room stereo" has been hit and miss for my parents to the point where I regret setting them up with a Chromecast-supporting AV receiver and Google Nest speaker. They now just use the regular remote and listen to the station via FM.

If you're unlucky enough to be bilingual, it's a complet joke. It's a real shame, because it makes this function useless for the people who need it most, those who can't type easily on a phone (visual challenges), or have difficulty transcribing names in a foreign language.

It could help millions of elderly people stay independent a little longer. But no instead, they were trying to sell us generative AI and photo filter.

The weird thing is that they just don't seem to care about improving it. WhisperX can process my speech just fine, but Google assistant can't. That's the lowest bar to clear and they don't need to invent any new tech for it - there's a number of solutions ready to go right now.

> If you're unlucky enough to be bilingual, it's a complet joke.

Yep, it's just pain :D I'm native portuguese speaker but I navigate the english web.

Hey siri, read notifications (portuguese name bad written) sent message <random portuguese read as english>.

or:

E aí siri, lê notificações (read notifications in portuguese) Notificação de Twitter, <english post read as portuguese>

For me, this recently started working! Do you have both Portugese and English configured as languages in your iOS settings? I believe that that's what motivates Siri to try and interpret a message in the other language as well when reading notifications.

hummm, found the "read message" language setting... I doubt it it will work properly but will try

Thanks :)

I dont know, it works perfect for my (non-native) accent even in the noisy car environment with a super-slow car unit. It's really good

Google's voice assistant is leaps and bounds better than next best thing available on a cell phone, Siri.

It's much much worse if you're trying to call someone without an Anglo-Saxon name too. It just outright refuses to call anyone I know with a Thai name, regardless of how I pronounce it.

The same goes for French and Romanian, I call it Starwars name problem. You have to make some sort of strange variation like the name in Starwars so that the device recognises it.

Yeah, I think the LLM for everything crazy is stupid, but I do wish there were a little more of it in the Google Assistant. Then I could finally say "Text Bob 'Home in 5'" and not have it reply "Sorry, I don't have a 'Home' number for Bob."

Let's wait until it actually ships and if it'll work offline. Last few times Google sold Pixels on dedicated AI hardware it wasn't even used and instead offloaded to their cloud services, making them slow and often blocking app UI until it's either finished or timed out.

Not only does it free the device manufacturers but it's something they can offer to app developer for free without costing them anything. Personally I think this is what will be the biggest impact (When/If they open raw "api" access to 3rd party devs). There are whole slews of apps that could make of that in small or major ways if it's not something the developer has to price into the app/subscription (especially with usage being something that can vary greatly user-to-user).

Some random examples I just came up with:

* Podcast app does STT for transcriptions/"subtitles" as well as episode summarization (Even cooler if the podcast provides chapter markers)

* Games getting new and interesting types of NPC's, level generation, etc

* Filters that do more than keyword matching (for news/social apps). I can technically filter out political Spam SMS messages but only with regex/keyword matching.

* Catch-up on 3rd party chat app conversations

I know LLMs will make plenty of mistakes and are far from perfect but there are things they can do very well that are very hard or expensive to do (sometimes just for indie-level devs but also for small/medium businesses). To be able to do those for free and offline would be a gamechanger.

> On-device inference improves not just its latency

this is a nuanced thing. sure you save the network hop, but you are running inference on a much more underpowered model on an underpowered device. at the end of the day you want to get the lowest time to first token and highest tokens per second. on-device optimizes TTFT at the expense of Tok/s. i think there are many usecases where Tok/s dominates in overall latency considerations.

If you watch Google’s keynote, the majority of LLM inference they showed happens on their cloud. (Also called private cloud to borrow apples term)

They actively call out only a few areas that run locally.

Almost all their summary features are in the cloud for example. Unlike Apple, they haven’t figured out API to let apps vend data to their AI system.

It's good for privacy so media companies probably don't want you to know it exists.

How useful is it really? It s a novelty, and as language are going to get truly useful and reasonable, the small llms will be nothing but a novelty item that drains the battery.

They don't say it explicitly that it's running LLM on phone. Built in Gemini could just mean pre installed and configured app that works with a connection.

tokens per second is normally the biggest cause of latency, so amortized I would bet a decent LLM on the phone is still slower than groq/aopenai etc

Reply on news.ycombinator.com