The pelican benchmark is exactly what's wrong with hiring in technology.
It's got nothing to do with what most people actually do when they're working - just like most job interviews which ask you to draw a pelican as their way of assessing you.
It's got nothing to do with what most people actually do when they're working - just like most job interviews which ask you to draw a pelican as their way of assessing you.
AI companies claim their products are generalists though, and that they can do a good job on anything you give them, so you can't say what people will be doing with it. "Generate an SVG of an bird on a bicycle" is a corner case certainly but if a candidate interviewing for a role claims they can handle the corner cases then it's totally fair to assess them on that.
Besides, if you move up one layer to "how good is AI at generating valid SVG markup of non-obvious things", pelican on a bike is actually a good test.
Because of this, I presume the Pelican has been in the training data for at least a year+.
The models are very useful, I am afraid they have fundamental limitations though generalizing (it is just hard to evaluate effectively). So it will just be whack-a-mole "can your model do X", and there will always be a new X.
So then add a dash of cybersecurity and medical use and that's basically it. No "closer to AGI" advertising. I'd say the 2026 development has in fact been the opposite; optimizing AI for niches where there is most potential for profits and that your description died in circa GPT-5 era.
In fact, this problem (for this test) is also stated by the pelican test author:
"The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.
So don’t go using pelicans to compare models!"
Empirically, they have something very much alike to the human "g factor" - a shared pool of "general intelligence" that all tasks benefit from.
When a "make it bigger, train it harder" upgrade like Kimi K3 or Mythos 5 drops, the performance rises on every metric. Not just the "headline benchmarks" like Mythos and coding/cybersecurity, but also things like literary analysis - which has nearly zero economic value, and isn't commonly post-trained or benchmarked for. And companies keep encountering things like "our carefully trained specialist model with lots of in-domain training on expensive closed datasets just got leapfrogged on our internal benchmarks by a next gen off the shelf generalist".
You can go hard on benchmarkmaxxing post-training, and you can burn millions of GPU-hours on coding RLVR. But, by the very nature of LLMs, a lot of the performance gains in flagship models are broad and domain-inspecific.
"Stiff prose" is more of a "style" thing than a "capability" thing. No one cares about how good an AI is at things like long form creative writing, because that's the opposite of a profitable field. All of LLM behavior is routed through text, so it's very easy to perturb "writing style" by some training elsewhere. Regression evaluation is hard. And the writing-specific post-training LLMs get is usually just cheap RLAF, with all the usual RLAF degeneracy.
Thus, we get the "default styles" that suck from a "creative writing" standpoint. A lot of that is just "what sounded good to the previous generation of LLMs" - and, unlike human readers, LLM evaluators don't get bored from seeing the same cliches repeated 9000 times across 9000 different instances of generated text. Humans tend to update over time from "this sound cool and punchy" to "this is generic AI slop", but RLAF evaluators stay at step 1. What little human-guided optimization this gets is aimed at "copywriting, marketing blurbs, punchy short-form" - and it shows.
You can do a lot there with some aggressive prompting, but the default writing styles suck, and I frankly don't expect that to change soon. No one cares enough to change it.
Pelicans? Used to be a decent proxy for "general model capabilities that no one would benchmaxx for" - a way to probe for that elusive "LLM g factor". Now that it's a known metric, it's very gameable. But it was pretty solid while it was novel and obscure.
* operates an absurd prompt
* involves SVG coding knowledge, generates a source code artifact
* involves world knowledge (what is a pelican? What is a bicycle? What does each do?” How are each constructed?”)
* when rendered, the coding artifact expresses an image that makes sense to us perceptually, including color and spatial relationships
* different models and settings have different output so it can be used as an evaluation scheme
That said I wouldn’t choose a model based on this! Just like some brain teaser shouldn’t determine employment eligibility.