Hacker News Reader

Comment by simonw | Hacker News Reader

For anyone who followed yesterday's Gemini 3.7 Flash pelican which rendered in Safari but not in Firefox or Chrome... https://news.ycombinator.com/item?id=49289112#49290012. - that turned out to be my fault, not the model.

My SVG rendering software was stripping some attributes. Here's the Gemini 3.7 Flash pelicans in the fixed renderer: https://tools.simonwillison.net/markdown-svg-renderer.html#u...

View on HN

Perhaps this isn’t a new observation but the problem with LLMs is very clear with these. It’s a nice microcosm. The LLM will draw a fish companion (unprompted!) with a nice gradient but won’t get the pelican’s feet right.

It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.

Or a matter of having limited time but spending it thinking about the wrong thing. Which, not to anthropomorphize it, but it's not like humans don't do that, or at least I do.

I'd like to see a human create a better pelican SVG without being able to look at the results. LLMs are language models, not (natively) vision models*. That's why it's a good benchmark: it engages LLM's logical reasoning in a way that we can check visually. The fact that they make errors which can be spotted visually, doesn't prove that LLMs are so far behind humans on language/logic tasks.

Besides which, "fundamental understanding" isn't a binary. Humans can also have a deep understanding of a subject and still make errors. (I do, anyways.) I do not think "fundamental understanding" is itself well enough understood to say definitively that LLMs do or do not have it.

*Some models have multimodal capabilities but these are separate weights, I believe it's only engaged in processing images. From the reasoning trace, the model never rendered the result -- only instructed the user to do so. I welcome correction if I'm wrong, I only have a surface-level understanding here.

AFAIK the only well-known LLM with no "conventional" multimodal encoders required for audio and images is Gemma 4 12B.

Hmm, doesn't seem to translate to an especially coherent pelican: https://xcancel.com/TeksEdge/status/2063108620842356970

Have you seen what happens when you ask 500 humans to draw a bicycle? No pelican, nobody riding it, just the bike.

https://foerstel.com/memory-bikes/

I suspect if you showed one human a million photos of a bike they might get it right.

It is interesting that it went with something like:

"pelican = ocean bird = nautical theme = this means I should put a captain's hat on the cartoon Pelican"

I think in some sense it understood the absurdity of a pelican riding a bicycle as intentionally nonsensical, so it went with a cartoon theme for the design and the shading. Worked out really well.

Damn That's very good

>> Most importantly, the pelican has one leg on each side of the bicycle - that's very rare.

How can you say "each side"? I don't see any z-ordering between bicycle frame and legs.

Because, if you look at the image you can very clearly see that the frame and relative components partially occlude one leg and not the other.

The model is presumably aiming for an acceptable visual representation, not trying to produce z-ordered components and a lost of any other random requirement people might come up with. The task is to show a pelican riding a bike, not produce a technical design that is layer order correct, after all.

Also:

> Near leg: from (300,245) to (356,408): thigh+shin as a single slightly bent line: M300,245 C 310,320 330,370 354,406. Stroke #F2953F width 12, linecap round. Far leg: from (320,250) to (404,452): M320,250 C 350,330 385,410 402,448. Stroke slightly darker #E0802F (behind, drawn before near leg but after bike? Pelican legs are in front of frame? Pelican is drawn after bike, so legs overlap the frame. The far leg ideally would be behind the frame, but in flat cartoon this is acceptable — or draw the far leg before the pelican body but after bike; overlaps the red frame.

Gemini is worse

Holy crap that is above and beyond

But, not animated like the Qwen 2.4T one you showed, so minus points for that

Animation was not requested.

Reply on news.ycombinator.com