Hacker News Reader

Comment by simonw | Hacker News Reader

Here are 18 pelicans - six each for Luna, Terra and Sol at the six different reasoning effort levels (plus the price to generate each one): https://static.simonwillison.net/static/2026/gpt-5.6-pelican...

Or if you want to see some in 3D, OpenAI featured a pelican riding a tricycle, bicycle, pony and another pelican in their livestream this morning: https://www.youtube.com/live/Wq45rvPGNHs?t=1070s

View on HN

Time to dump this test. Probably not a coincidence every version has the same rolling green hills, gradient blue sky, sun in the corner, etc.

On the one hand: yes, pelicans on bikes are definitely in the training set at this point.

On the other hand: the test is clearly not saturated, given that you can see a clear difference in output at the various reasoning levels / model versions.

I sort of agree, but within the same model I expect the reasoning effort to be reflected in the quality of output and that's basically how it played out. When you're comparing different models, then it's just who benchmaxxed the best and there's not a lot of value there.

But that is my point: if benchmaxxing was all the labs were doing, then surely the dumber model could/would have equivalent performance? Rather than noticeably worse perf on a (somewhat trivial to game) test.

Goodhart's law.

I don't know. If they were training on this, I feel like they would be able to get the shape of a bike frame right; it's a pretty simple polygon, and a lot of the bike frames that are getting generated would be impossible to steer.

I mean, humans can't draw bikes!

https://themagnet.substack.com/p/why-is-it-so-hard-to-draw-a...

I partially agree, but in this case it kinda illustrates that it may not be worth using Terra on any reasoning level below high; those are some awful penguins on bikes.

Waiting for the AMA on Reddit "Ten years ago I was responsible for the pelican department at OpenAI, AMA"

I think the 'pelican test' is becoming useless. It's been around long enough that now I'm sure good examples are in the training data, and hell they might even do some hand tuning to make it do a decent job since they know people will ask about it.

But either way, with no real way to visualize the result of the text it starts with - it will always be stabbing in the dark. It can't understand conceptually what any of it should look like and then refine the SVG to improve it gradually. It just throws darts at a wall and hopes it comes out alright.

I think it's still useful in a "hello world" sort of way. It means you actually tried the new model.

Honestly that's the main value I get from it myself - making a pelican means I have to figure out API keys and how to talk to the provider, or how to run it locally for the local models.

Pelicans, maybe, but the point is to measure how good the "internal visualization" abilities are. Throw curveballs, like a unicorn with a duck bill serving coffee at a basketball court. An elephant playing a piano while its trunk swings a baseball bat at a tiny alien spaceship buzzing its head.

Have them use tikz instead of svg, or have it write code that moves the cursor and draws the thing in paint.

Compositionality and visualization are generally much, much better at each new generation / release cycle.

It's fascinating how well models have internalized visualizing things without actually having joint embeddings / broad multimodality.

Still fun to watch models trying their best :). I think "money spent" metric in this test is becoming the most interesting to watch. It is like looking at RT cost of "Hello, world!"...

What's strange with this is the prompt "Photorealistic photograph of a pelican riding a bicycle down a coastal boardwalk, wings gripping the handlebars, webbed feet on the pedals, large orange bill, detailed feather texture, golden hour lighting, shallow depth of field, shot on a DSLR with 85mm lens, natural motion blur on the wheels" produced, well, exactly what I asked it for. I wonder if I tell it then to make it SVG ...

https://chatgpt.com/share/6a5009de-fff8-83ea-98ff-0da17d1d04...

Cool. I still find these a useful visualization of some the qualities of llms. Even if they did train for [animal] on [vehicle] svg, it's still nice to see at a glance how the different models and reasoning levels perform. Lunar misses part of the frame, except on max reasoning. While most of the others have a mostly correct bike at all reasoning levels.

I once used something like karpathy's auto-scientist to mutate the prompts and rank them with a vison model. Some of the winners where pretty neat. I think they have a lot more style than the gpt-5.6 ones. https://xcancel.com/xundecidability/status/20449185674144196...

people are saying this is benchmark is saturated but all of these have occlusion issues, even sol max.

A skilled human artist wouldn't have both legs in front of the bike, or a single straight line representing both leg's crank arms.

Yeah it makes no sense at all to dismiss the test, when even the very best examples are noticeably below what a skilled teenager could produce.

Dead internet theory? Semi-random parroting by real people? Or something else.

Is the direction of the pelicans encoded in your prompt? Curious why they are all left to right with the exception of terra xhigh.

History moves left to right.

Cultural bias.

At what stages will models start to internally reflect the drawn SVG and automatically fix their own mistakes?

I assume multimodal models can do it already do it today if constantly asked "make it better"

I haven't tried this in a few months, but last time I tried a loop that rendered the pelican and asked for improvements the results were actually quite disappointing. Be interesting to try that again against GPT-5.6 at Claude Fable 5 though.

Please do!

The quality of sol on effort=none makes me think this test is saturated or they are benchmarkmaxxing this exercise.

Ok, I'll never use max effort again on OAI models..

Is that... an x-rated, censored pelican?

They said in the AI community, a pelican riding a bicycle is a good test to measure effectiveness of the model, wondering if they were referring to you, or is it really a standard in the AI community ?

Also would be good to have a tool where users can select models and instantly see each model's generated pelicans. That will make it easy to compare the output of different models.

Simon did start the pelicans on bicycles as an SVG, but I think it's more of a fun goofy thing to see how the model performs at. I don't think it has a direct correlation to a model's performance though.

I'm waiting for the day that the "generate a Pelican" test comes back with a SVG-art like illustration of a Pelican equipment case, like a model 1620 or similar.

https://www.google.com/search?client=firefox-b-d&q=pelican+1...

Surely "how to draw a SVG pelican on a bike" has made it into the training data by now ...

And yet... only one is any good.

Surely this comment is literally on every new model release post.

It's part of the pelican tradition at this point.

I eagerly await the models replying with that: "I'd be happy to create a pelican riding a bicycle, but just a note that this might already be in my training data. Simon."

Someone told me that a model said "oh, the classic" when they asked for this recently. I think that was Opus 4.8.

If that was the case in a non-trivial way you'd see mode collapse, but you don't, they come out differently.

It's because all the frequent comment that this pelican is in the trainings set now also got into the trainings set and models adapt. /joking (I hope)

Thank you Simon! Luna is surprisingly decent across all reasoning levels.

I think all of Luna's are bad. The only decent one is sol @ xhigh. Even sol @ max is weird. Sol @ high and @ medium are ok, and every other single one across every model is bad.

Strong disagree, but to each their own. For sol I really like how only medium uses the wings on the handlebars to ride the bike. For all the other sols the pelican evolved a new set of arms separate from the wings.

something is wrong with Terra model series, most pelicans, except Max, looks bad

Seems to match the pareto frontier on Artificial Analysis as well. Terra is nowhere on it.

gpt-5.6-sol Max pelican didn’t skip neck day

gpt-5.6-sol x XHIGH is my favourite

Apparently plus users do not have access to Sol, so I'm really worried about the ugly Terra Pelican.

somehow Terra really struggles here even compare to Luna.

max effort sol clearly over-engineered

AI really sucks at bicycles...

LLMs really suck*

I like Terra High the best. That pelican is utterly yoked.

terra is just weird. in this nothingburger test, time nor higher costs seem to not strongly correlate with the aesthetics.

this looks like the same shit from 4 years ago. give it up.

The first time I did this was actually less than two years ago - in October 2024 - and it's fun seeing how much better they've got since then: https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/

Absolutely, the change in quality over time is a great yardstick.

Nice to see you did the quality level comparisons and did three passes.

I've been using that technique myself on my image gen reviews[1] and it also works well in presentations and for personal study.

[1] https://generative-ai.review/2025/12/beast-mode-activated-op...

Reply on news.ycombinator.com