Hacker News Reader

can llama 3.3 vision do things like "there's a textbox/form field at location 1000, 800 with label "address"" ?

I did a quick and dirty prototype with Claud for this, but it returned everything with an offset and/or scaled.

Would be a killer app to be able to auto-fill any form using OCR.

Were you using claude’s computer mode? It can do this

No, I used the regular Claude which can also (somewhat) do this and uses the same image processing backend as "computer use" as far as I know (source: Antropic CEO interview with Lex Friedman recently).

Computer use is also not very good at it (often mis-clicking for example).

I'm guessing this will work flawlessly within 6 months to a year or so, but it doesn't seem ready yet.

Reply on news.ycombinator.com