Hacker News Reader

For anyone who followed yesterday's Gemini 3.7 Flash pelican which rendered in Safari but not in Firefox or Chrome... https://news.ycombinator.com/item?id=49289112#49290012. - that turned out to be my fault, not the model.

My SVG rendering software was stripping some attributes. Here's the Gemini 3.7 Flash pelicans in the fixed renderer: https://tools.simonwillison.net/markdown-svg-renderer.html#u...

Gemini is worse

Damn That's very good

Holy crap that is above and beyond

But, not animated like the Qwen 2.4T one you showed, so minus points for that

Animation was not requested.

Perhaps this isn’t a new observation but the problem with LLMs is very clear with these. It’s a nice microcosm. The LLM will draw a fish companion (unprompted!) with a nice gradient but won’t get the pelican’s feet right.

It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.

Or a matter of having limited time but spending it thinking about the wrong thing. Which, not to anthropomorphize it, but it's not like humans don't do that, or at least I do.

I'd like to see a human create a better pelican SVG without being able to look at the results. LLMs are language models, not (natively) vision models*. That's why it's a good benchmark: it engages LLM's logical reasoning in a way that we can check visually. The fact that they make errors which can be spotted visually, doesn't prove that LLMs are so far behind humans on language/logic tasks.

Besides which, "fundamental understanding" isn't a binary. Humans can also have a deep understanding of a subject and still make errors. (I do, anyways.) I do not think "fundamental understanding" is itself well enough understood to say definitively that LLMs do or do not have it.

*Some models have multimodal capabilities but these are separate weights, I believe it's only engaged in processing images. From the reasoning trace, the model never rendered the result -- only instructed the user to do so. I welcome correction if I'm wrong, I only have a surface-level understanding here.

AFAIK the only well-known LLM with no "conventional" multimodal encoders required for audio and images is Gemma 4 12B.

Hmm, doesn't seem to translate to an especially coherent pelican: https://xcancel.com/TeksEdge/status/2063108620842356970

Have you seen what happens when you ask 500 humans to draw a bicycle? No pelican, nobody riding it, just the bike.

https://foerstel.com/memory-bikes/

I suspect if you showed one human a million photos of a bike they might get it right.

It is interesting that it went with something like:

"pelican = ocean bird = nautical theme = this means I should put a captain's hat on the cartoon Pelican"

I think in some sense it understood the absurdity of a pelican riding a bicycle as intentionally nonsensical, so it went with a cartoon theme for the design and the shading. Worked out really well.

>> Most importantly, the pelican has one leg on each side of the bicycle - that's very rare.

How can you say "each side"? I don't see any z-ordering between bicycle frame and legs.

Because, if you look at the image you can very clearly see that the frame and relative components partially occlude one leg and not the other.

The model is presumably aiming for an acceptable visual representation, not trying to produce z-ordered components and a lost of any other random requirement people might come up with. The task is to show a pelican riding a bike, not produce a technical design that is layer order correct, after all.

Also:

> Near leg: from (300,245) to (356,408): thigh+shin as a single slightly bent line: M300,245 C 310,320 330,370 354,406. Stroke #F2953F width 12, linecap round. Far leg: from (320,250) to (404,452): M320,250 C 350,330 385,410 402,448. Stroke slightly darker #E0802F (behind, drawn before near leg but after bike? Pelican legs are in front of frame? Pelican is drawn after bike, so legs overlap the frame. The far leg ideally would be behind the frame, but in flat cartoon this is acceptable — or draw the far leg before the pelican body but after bike; overlaps the red frame.

Totally agree. I actually didn't know you already got to it and I ran it on mine. It even added a scarf lol

https://imgur.com/a/IgYKnmk

Took almost 90 minutes on my Threadripper Pro, and feels well worth it.

28k tokens from the Q8 model

And yours got the chain!

What's the result if you ask it for a "pelican equipment case" ? I've been trying the anti-bicycle pelican on some LLMs and the results are much more varied than the bicycle prompt. Some have very different ideas of pelicans (you'll get a small case with one DSLR camera in it, or a long rifle case, etc). You'll also get cases that are isometric view, or flat plane view from the front, or open or closed.

It did a pretty outstanding job on that: https://tools.simonwillison.net/markdown-svg-renderer.html#u...

This took 56 minutes! - but it didn't have exclusive use of LM Studio, I ran a few other test prompts while it was working.

  llm -m lmstudio/qwen/qwen3.8-27b 'Generate an SVG of a pelican equipment case'

That is pretty good! It's interesting it chose orange, because that's actually a somewhat rare color for pelican cases. I would say 95% of the ones going around the world right now are either black or FDE (flat dark earth) tan colored. There are some orange ones, for sure, if you google image search "orange pelican case", people use them for emergency/first aid supplies or electronics. But only the smallest ones are available in orange.

It is also worth noting that it shaded the pelican case like it was a product photo taken in a studio, there's a virtual light source above and to the top left corner of it, then shading deeper color of orange as the body area progresses towards the lower right corner.

What are the chances that the colouring of actual pelicans is bleeding in to the prompt. Seems it could be that?

Doubtful? Without knowing anything about how it was trained, if you google image search "pelican bird" they are predominantly white in color. Though the bill is certainly orange, different shades depending on camera and lighting conditions. The SVG attempt looks more like an attempt at the product photos if you search "pelican case orange".

Need more tests of 'pelican bird standing in pelican equipment case' ;)

Or to make it even more tricky, since Pelican cases will float when closed, something like "generate an SVG file of a Pelican bird floating in the ocean next to a Pelican case".

I've been getting ads in my Instagram for http://colorcase.com. They add colors and wheels to cases and I'm sharing them because you're right, most of them aren't orange. But they could be!

Wow, those are expensive. I've been buying pelican 1510 and similar from eBay and b&h used/refurb. The harbor freight pelican knock off that is their 3800 and 4800 product numbers are also not bad. It's like 85% as good as a real pelican for 40% of the price.

it's just money and you wanna look good. You're worth it.

Not too bad, on my 5090 I got ~160tokens/sec and it took 3m20s to generate a pelican using Ninfer which someone else mentioned. I don't know why I didn't get any sweet sweet seagulls in my background, even on xhigh reasoning effort.

https://gist.github.com/hansale/ed9e73fe35165a58ea2af6b1632a...

I've done a few variations, I've been impressed with all of them. My favourite so far has been "Generate an SVG of a turtle flying a kite", result: https://imgur.com/a/bdKJPV4.

Some will say conflating flying and flying a kite is bad, but I like it.

Have you thought about running a second tier of the Pelican benchmark where you see which model makes best pelican on lowest or no reasoning settings? I think that'd be pretty interesting and might help highlight which models have a baseline capability - even within those I'd imagine the token usage would vary wildly and might give some indications on verbosity as well.

> Bicycle is the right shape. Pelican beak is excellent. Nice background.

Relative to other results I agree. But on an absolute measure, there is not a single element in the current bicycle that is real-world accurate and many elements are omitted or non-physical (eg, the transparent seat tube top, entire lack of a head tube).

Consider a series of followup benchmarks.

With a fresh context of the LLM under test, ask it to generate a list of findings for how the pelican-on-a-bicycle SVG that was produced is inaccurate compared what the real world scene might appear, accepting for the limitations of SVG as a medium. Then, feed back the list of findings to the original context for a second try. The benchmark can stop here by humans looking at the result and forming their own conclusion.

Next phase is to repeat the analysis phase using the 2nd context to determine what findings were satisfied and what new inaccuracies are found. These two differences can form a second benchmark.

Last phase is to iterate with the goal to drive the number of findings to zero.

I was just updating a Proxmox cluster from v8 to v9 and had time on my hands and 15-30 mins after the models appeared for general download I had an initial pelican effort in .svg format.

I switch off reasoning by default and for starters I ran 3.8 the same way as 3.6 on vllm on a DGX Spark box. Open web UI prompted.

My word! It looks rather decent. Much better than previous efforts.

It's definitely trained on generating SVGs. Not that they trained on the pelican prompt itself, but probably a more general RL step that has it generate SVG and a stronger vision model judges it.

It's up to you if that counts as benchmaxxed.

Generation of SVGs is (obviously) a generally useful task.

Using a better quant Q6_K (Unsloth's) compared to your quant Q4_K_M (LM Studio's) yields this: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

Chains exist. Red scarf is proactively added. ("Maybe a scarf blowing in the wind for charm!) No hands/wings, though.

Generated 30.2k tokens in total and took 52 mins on M5 Pro in low power mode (it will possibly take less than half of that in auto energy mode).

Even with "low" thinking effort and Q4 it makes reasonable pelican, only 4k tokens and half a minute on 4090: https://imgur.com/a/yd5rNCp

If I'm doing the math right, that's like 17 t/s? I haven't played with Qwen 3.8 yet, but that seems really slow for a 27B on an M5 Max with sufficient RAM to hold it in memory.

this sort of test surely only works when it's a brand new concept/prompt. like the exact same models trained newer data will surely get better at putting this together over time

Try it out with your own prompt, I suspect you'll be surprised.

I asked it to make an animated svg of a pelican doing tricks on a bmx bike and it did. I then asked it to turn that into a browser game. This model fucks.

Ask it to do a zebra on a jetski to escape the benchmax.

what type of laptop runs something like that? using mac or egpu?

See, like my comment before. We don’t need the biggest/highest/frotiest model.

I think the model got trained only on pelican to get there

We should start asking it for an Albatross instead.