Hacker News Reader

Comment by notaustinpowers | Hacker News Reader

I'm sure it's part of their ML with the newer iOS but this does make me happy to think of the AI running on the hardware. Yesterday when driving I got a message and usually it would just say "Jane says 'blah blah' and sent a photo." But last night Siri in CarPlay said "Jane says 'blah blah' and sent a photo of a dog in a blanket on a couch". And when I checked the photo they sent it was exactly that.

If it's E2E encrypted, I don't think it'd be able to identify that during transit, so it had to have been locally done.

Thought it was interesting, and a nice QoL update. Excited to see how this can grow if they introduce more things to be processed with AI on the hardware.

View on HN

I love how Apple obsessively does things on the edge, on the device itself rather than on the cloud.

Sometimes my lowly iPhone X gets super hot during charging. Next morning I either get the facial recognition updated, a new memory generated or something else has happened. It's nice, confidence evoking even. The features you like are run on your device, with your data not leaving your device, on its free time.

This is the AI I like. Personal, confined and private.

They're in the best possible position to innovate on the literal edge, because they've got a fleet of billions of devices that mostly auto-update. As hardware advances, Apple will have the clearest path to deploying truly personal and locally hosted AI.

> truly personal and locally hosted AI.

If we can combine this with a neuro bill of privacy/rights [0] then the future might not be as bleak as it seems.

[0] https://www.cirsd.org/en/horizons/horizons-winter-2021-issue...

> Sean Carroll & Nita Farahany on Ethics, Law, and Neurotechnology

https://www.preposterousuniverse.com/podcast/2023/03/13/229-...

And because it's their business model, you don't have to worry about it changing.

They do stuff on-device because they want to keep selling expensive devices.

It's fantastic for them that they sell the hardware, unlike Google/OpenAI that have to spend on hardware.

More so these days you buy some expensive hardware but keeping subscribing to apps and buy TV shows and cloud storage from them due to convenience. And then you buy the next model after a few years because why would you change a working formula for some Samsung/Google android BS that won’t be as clean….

> If it's E2E encrypted, I don't think it'd be able to identify that during transit, so it had to have been locally done.

Yes. Image description has been done on-device for quite a while now, and it is great. You can look for pictures using a description, and you can look for any text in any picture as well. It works very, very well.

Yeah, but I agree with the parent; I think something has changed recently with Siri. The other day I asked it some questions as a joke that I knew would just say "here's what I found on the web", but it answered them. It made me wonder what they're changing under the hood.

Yeah, apparently well enough to run a web service off of iPhone image recognition!

https://findthatmeme.com/blog/2023/01/08/image-stacks-and-ip...

That is seriously impressive!

> It works very, very well.

I had always assumed that search box in Photos.app would be useless, but once in desperation I typed something in and it found exactly what I was looking for (a photo of one of my kids' passports I knew I'd taken but could not for the life of me remember where).

Now I use it more and more, it makes my tens of thousands of photos dramatically more useful to me. That time we at at the restaurant with the big shrimp platter? The airbnb we stayed at where I took a picture of the cool coffee table? All right there.

Maybe it’s not especially amazing in 2023 but I totally agree. Finding random documents you snapped a photo of because you’re looking for some text that happened to appear on the edge of the page in a tiny font, it’s really useful. L

I’ve used the feature to find my cars VIN, passport numbers, random places etc.

My Samsung also does that, though I wouldn't say it works very well.

Same for Samsung devices with the official Samsung photo app

It is amazing for accessibility users too, if you turn on VoiceOver, the Camera app and the Photos app tells you the description of what’s currently on display. And yes, it has been a feature for a while.

The magnifier app has that and reads the results live as you point the phone at things.

It also has other accessibility features like helping you find doors.

You may need to turn something on in settings before all of this shows up though, I don’t remember.

Apple photos is wonderful for this. I can search for “cows” and I get photos of cows. All on device in the middle of a mountain with no internet 50 miles in any direction.

Plus I believe it indexes the text in images too. So if you remember something on a sign that you took a picture of it can find the picture without you having to know where/when it was.

It was! And it works surprisingly well. It won’t describe movies/animated gifs or images that are very small though.

It’s new for iOS 17.

Reply on news.ycombinator.com