Apple researchers develop AI that can 'see' and understand screen context

Many a time I’m surprised by how good Siri’s description of images is. I let Siri speak out some notifications when I’m wearing my EarPods:

- $person sent a picture of a group of people waiting at an airport.

- $person sent a screenshot of a social media website with an image and a caption.

- $person sent a picture of a dog sleeping on a wooden floor.

All of this to say: I’m not surprised they’re investing further into this.

We (Google Research) have a similar work that we recently published: https://blog.research.google/2024/03/screenai-visual-languag...

jwz has a rant (not linked here because he presents a special page for hacker-news referral links) about people sending him screenshots of errors instead of typing the error text into an email. I kind of understood where he was coming from (why encode text as an image, that requires more congnitive load to decode than the original text) but I actually frequently see error screenshots that contain more contextual detail (the OS, what extra software is on the machine, etc) that can help in understanding a bug.

To me, having systems that can do developer-level triage from screen contexts would be pretty valuable although in principle, it shouldn't be needed.

This is interesting in the context of the Rabbit R1-style AI. I forget what they called it, but it’s learning how to ‘drive’ your computer like you do vs. interacting with an API.

So to book a flight it just pretends to be a human and uses the website.

I see a fascinating future where you can teach an AI how to pay your credit card bill. Because I can explain it to you trivially:

1. Log in to credit card provider website.

2. Look at {this number} on the screen. That’s what I owe this month.

3. Log in to my bank.

4. Click Pay > choose ‘credit card’ from my ‘stored contacts’ > paste that amount.

5. Wait for 2FA code to arrive via SMS.

6. Paste 2FA code.

Yes please. I’d like to be able to script my life like this, such that the AI can adapt when my credit card provider changes their CSS class names or whatever would break my fragile Node script.

Can it also understand and solve captchas?