Hacker News Reader

I keep seeing “Claude image understanding is poor” being repeated, but I’ve experienced the opposite.

I was running some sentiment analysis experiments; describe the subject and the subjects emotional state kind of thing. It picked up on a lot of little detail; the brand name of my guitar amplifier in the background, what my t shirt said and that I must enjoy craft beer and or running (it was a craft beer 5k kind of thing), and picked up on my movement through multiple frames. This was a video slicing a frame every 500ms, it noticed me flexing, giving the finger, appearing happy, angry, etc. I was really surprised how much it picked up on, and how well it connected those dots together.

I regularly show Claude Code a screenshot of a completely broken UI--lots of cut off text, overlapping elements all over the place, the works--and Claude will reply something like "Perfect! The screenshot shows that XYZ is working."

I can describe what is wrong with the screenshot to make Claude fix the problem, but it's not entirely clear to what extent it's using the screenshot versus my description. Any human with two brain cells wouldn't need the problems pointed out.

This is my experience as well. If CC does something, and I get broken results and reply with just an image it will almost always reply with "X is working!" response. Sometimes just telling it to look more closely is enough, or sometimes I have to be more specific. It seems to be able to read text from screenshots of logs just fine though and always seems to process those as I'd expect.

> Claude is definitely not taking screenshots of that desktop & organizing, it's using normal file management cli tools

Are you sure about that?

Try "claude --chrome" with the CLI tool and watch what it does in the web browser.

It takes screenshots all the time to feed back into the multimodal vision and help it navigate.

It can look at the HTML or the JavaScript but Claude seems to find it "easier" to take a screenshot to find out what exactly is on the screen. Not parse the DOM.

So I don't know how Cowork does this, but there is no reason it couldn't be doing the same thing.

I wonder if there's something to be said about screenshots preventing context poisoning vs parsing. Or in other words, the "poison" would have to be visible and obvious on the page where as it could be easily hidden in the DOM.

And I do know there are ways to hide data like watermarks in images but I do not know if that would be able to poison an AI.

Considering that very subtle not-human-visible tweaks can make vision models misclassify inputs, it seems very plausible that you can include non-human-visible content the model consumes.

https://cacm.acm.org/news/when-images-fool-ai-models/

https://arxiv.org/abs/2306.13213

Maybe at one time, but it absolutely understands images now. In VSCode Copilot, I am working on a python app that generates mesh files that are imported in a blender project. I can take a screenshot of what the mesh file looks like and ask Claude code questions about the object, in context of a Blender file. It even built a test script that would generate the mesh and import it into the Blender project, and render a screenshot. It built me a vscode Task to automate the entire workflow and then compare image to a mock image. I found its understanding of the images almost spooky.

100% confirm Opus 4.5 is very image smart.

im doing extremely detailed and extremely visual javascript uis with claude code with reactjs and tailwind. driven by lots of screenshots, which often one shot the solution

Using any plugins or skills for that work?

Claude Opus 4.5 can understand images: one thing I've done frequently in Claude Code and have had great success is just showing it an image of weird visual behavior (drag and drop into CC) and it finds the bug near-immediately.

The issue is that Claude Code won't automatically Read images by default as a part of its flow: you have to very explicitly prompt it to do so. I suspect a Skill may be more useful here.

I've done similar while debugging an iOS app I've been working on this past year.

Occasionally it needs some poking and prodding but not to a substantial degree.

I also was able to use it to generate SVG files based on in-app design using screenshots and code that handles rendering the UI and it was able to do a decent job. Granted not the most complex of SVG but the process worked.