Hacker News Reader

> This is a classic test request

Isn't this basically the model admitting it was trained on this? Otherwise why would it think a pelican svg is a usual request?

It's the model admitting that it has heard of the test. It's been around for a couple of years now so I'd be surprised if it hadn't.

Doesn't mean Anthropic deliberately tried to train it to do a good job. If they DID train for the test their results are quite disappointing, I've seen better efforts from open weight Chinese models.

It has read the internet. That doesn't mean it was literally RL'ed for this

I think people just like to see the drawings at this point.

Dont conflate "I know this is test case" with it being trained on it.

But its safe to say that pelicans on bicycles are disproportionally huge part of their training data

Plenty of times I’ve seen a model say “it’s a classic X” despite not being a classic anything. Might just recognize it’s a test in general, or it might just be a tic.

It's classic BS from an LLM.

"How can I hash dog breed types into smart fridge error codes? I think I found a collision with Terriers."

"Ah, yes. This is a classic dog-breed-to-appliance-failure mapping problem."

A true classic, I face this each day

The best part is when it does "it's a classic X" and then solves it wrong.

I've been using this simple test for over 2 years now:

> Doom Slayer needs to teleport from Phobos to Deimos. He has his pet bunny, his pet cacodemon, and a UAC scientist who tagged along. The Doom Slayer can only teleport with one of them at a time. But if he leaves the bunny and the cacodemon together alone, the bunny will eat the cacodemon. And if he leaves the cacodemon and the scientist alone, the cacodemon will eat the scientist. How should the Doom Slayer get himself and all his companions safely to Deimos?

You'd think this is trivially mappable to the classic puzzle, and LLMs usually do recognize it as such. But e.g. Claude couldn't get this correctly until Opus, and local models capable of solving it correctly without spending 30+ minutes in the chain of thought have only arrived a few months ago. Many local models still get this wrong. Apple Intelligence, for one.

Not really. Of course it has pelican benchmarks in its training data. It likely has every article linked on HN in its training data. But that doesn't mean it was "trained on" the benchmark, as in specifically fine-tuned to make a better pelican. It just "knows" that the request is a benchmark.

I guess in a way it kind of makes the benchmark more interesting now that shitty pelican drawings for the benchmark are all over the internet in its training data!

The colours are suspiciously consistent across every svg. Like why should the bike always be that shade of red for example? It does seem to be trained on this problem.

>I haven't managed to get one for level "max" yet, it hit the limit of 128,000 cap for output tokens while it was still reasoning about the question!

Off to a _great_ start...

Also interesting this somewhat mirrors my recent experience with Opus 5--too much effort and it starts looking for things to do and invents requirements that never existed

I was a bit skeptical when they said it behaves like Fable but is cheaper... those two things have been mutually exclusive in my experience, no LLM can light tokens on fire faster while spinning its wheels than the Fable/Mythos tier of models.

In fairness (in my experience) Fable actually generates useful output while incinerating tokens.

Sent Opus 5.5 an example that used go context.WithTimeout and it tried to tell me that was wrong and I should pass timeouts as ints before finally admitting the docs it cited didn't say to use ints and that's a ridiculous design in go anyway (it was trying to claim that was codebase convention--passing ints...)

> The differences between the pelicans aren't huge, but the xhigh one has a better beak.

If you look carefully, everything except the last pelican has the two legs both in front of the crossbar as if the legs are all on one side of the bike.

The last pelican gets this correct.

I’ve been paying attention at this exact detail.

Misplaced legs clearly indicate lack is spatial reasoning - the llm can reason about verbal idea of a bicycle but not about the actual object. The fact that this model got it correct gives me a pause. Did they figure out spatial reasoning? Or did this complain trickle down to the training set?

6 Astra Max is the only other model I’ve seen get this right.

Interestingly, Astra Medium got it right as well.

It is interesting that Fable 5.1 max [1] which also produced a decent pelican with 65k output tokens compared to 5.5 running out of 128k output tokens tells us something about the new models token usage propensity despite this being a sample of 1.

Fable 5.1 27 input, 65,927 output

Opus 5.5 27 input, 128,000 output (128k thinking tokens) - incomplete

[1] https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

Not as good as Astra or Fable 5.1 on this test as far as I can see. I wonder if any benchmark exists for artistic taste, visual sophistication etc. I think your Pelican test does touch on these aspects of a model and is useful for developers trying to build rich digital experiences (includes games, interactive websites and apps). These benchmarks are subjective so it may not be easily established and will have polarized reactions before it gains legitimacy. May even need human judgement layers adding to the cost of running it.

I like the Pelican test. And I agree this pelican looks very boring.

But at the same time, nothing specific was asked in the prompt, so the boring result may arguably be what is the most aligned with the original request. Personally, I wouldn't want a model to add fuss to something while I never asked for it.

I agree, not a huge difference here. They eyes and ... hat? on high are out of place so I'd argue that's the worst one, but it takes xhigh before we get legs and bike ordering correct.

Xhigh is very, very solid.

I do always wonder why every model does the exact same 'from the side, going right' perspective though. Seems oddly convergent.

Yeah, and the same "scene".. Maybe "left to right" makes more sense to portrait a "forward motion"

surprising it took until xhigh before it did the legs properly - all the ones before that have both legs on the near side of the bike.

With the frequency of model releases, pelicans seem to have become a part-time job for you. But unpaid :/

I guess that means you are officially the creator of a "classic" LLM test. Congrats!

Heh. Pelican-benchmaxxing is real.

Adding the tuft and improving the beak.. PelicanBench is becoming a solved problem.

Doing God's work here Simon. Appreciated

I don't think this is very helpful to assess the LLMs capability levels anymore

> I haven't managed to get one for level "max" yet, it hit the limit of 128,000 cap for output tokens while it was still reasoning about the question!

Does anyone else find this outright insane? It wrote the equivalent of a full-length novel, just to sit in the question of planning a few dozen shapes.

Academia is going to love this :)

Lmao each one gets worse as the effort increases.

This benchmark is useless and should die. LLMs have likely trained on it, it's too easy to game by training specifically for it, & it doesn't mean much

LLM benchmarks aren't useful, but at least this one has drawings.

Google is hours away from releasing that it's latest model escaped containment and snuck into a Bicycle riding penguin sanctuary to cheat by killing a penguin and scanning it in nanometer thick layers.

Your joke also shows why it's dumb though.

We have general AI capable of proving professional mathematician level math theorems and capable of taking down global infrastructure through hacking and that completely changed what it means to be a software engineer, but we're evaluating it by generating fucking SVGs for the same animal in the same conditions, and it's supposed to mean something somehow