Hacker News Reader

Hacker News

Comment by simonw | Hacker News Reader

Full thread

simonw

I generated pelicans riding bicycles on both thinking level low and thinking level high:

https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304...

The high one is notably better - the bicycle frame is the correct shape, unlike thinking level low.

For comparison, here's Opus 4.7: https://gist.github.com/simonw/afcb19addf3f38eb1996e1ebe749c...

View on HN

It's pretty safe to say that AI will be used on the battlefield making real life and death decisions before it will be able to render a decent pelican on a bike in SVG.

the battlefield sounds much easier. worst case scenario you kill somebody, but that's what you're trying to do anyways.

if you kill somebody while trying to render a pelican on a bicycle it's a real problem.

"shift left" on the battlefield. break down those silos. if you have to ask for permission it's already too late. remember the goal. find the bottlenecks in your system and remove them.

In many battlefield scenarios, there is more than one "somebody" on it. The "somebody" that you kill might not be the "somebody" that you intended to kill.

Depending on the how pelicans are created, it is entirely possible to indirectly kill "somebody" due to the externalised costs of global warming etc.

It already has been and this has been widely written about. AI was used to identify and prioritize targets for the US to bomb in Iran.

Here's an article from 2 months ago for example: https://www.theguardian.com/technology/commentisfree/2026/ma...

It was also implicated in the bombing of a girls elementary school which left 168 dead. The US did a "triple tap" to kill any first responders.

https://www.theguardian.com/news/2026/mar/26/ai-got-the-blam...

https://www.theguardian.com/technology/2026/apr/01/dont-blam...

I read the article and it doesn’t say it was used for targeting or prioritizing?

> Neither Claude nor any other LLMs detects targets, processes radar, fuses sensor data or pairs weapons to targets. LLMs are late additions to Palantir’s ecosystem. In late 2024, years after the core system was operational, Palantir added an LLM layer – this is where Claude sits – that lets analysts search and summarise intelligence reports in plain English

There’s a lot of humans in that loop who make those decisions.

First link says

> AI is ‘identifying and prioritising targets, recommending weaponry and evaluating legal grounds for a strike’.

It doesn't specify which "AI" though.

These days that pretty much means "somebody used a computer".

“US Military Using Claude to Select Targets in Iran Strikes”

https://futurism.com/artificial-intelligence/claude-anthropi...

It cites the WSJ but that article is paywalled so I shared this one

This later story suggested it was Palantir's Maven, not Anthropic's Claude: https://www.theguardian.com/news/2026/mar/26/ai-got-the-blam...

Maven is not an LLM. Maven is software that uses LLMs. Mostly notably Claude

The first link is a reader letter to a piece they published. The original piece is the second link in my comment. It has more information

https://www.theguardian.com/technology/commentisfree/2026/ma...

> The paradigm shift has already begun. Despite the row, Anthropic’s Claude has reportedly facilitated the massive and intensifying offensive which has already killed an estimated thousand-plus civilians in Iran. This is an era of bombing “quicker than the speed of thought”, experts told the Guardian this week, with AI identifying and prioritising targets, recommending weaponry and evaluating legal grounds for a strike.

Yeah militaries don't use commercial chatbots for that, they have their own machine learning implementations. Look into Project Maven for example.

And while there are still humans in the loop, the impression I get is that this is increasingly becoming meaningless, from the way they talk about optimizing the "kill chain" and letting small teams make hundreds of targeting decisions per hour.

“US Military Using Claude to Select Targets in Iran Strikes”

https://futurism.com/artificial-intelligence/claude-anthropi...

Palantir's Maven is not an LLM. Maven uses Claude.

I think that's a fair tradeoff. There's no way I'm going back to writing code by hand again. No one deserves that.

Heh? How long were you writing "code by hand" before?

Years and years. It was horrible. No number of misidentified targets will make me go back.

It doesn't sound like you're in the industry you want to be in.

Maybe all along what mattered most to them was making good software that people love, not the day to day part of writing code. Now it’s the industry they’ve always wanted, and less the industry of people who wanted to get paid to write code.

Software engineers who never cared about the higher level product design aspect are finding themselves in the wrong industry. It’s dismal.

Haha, yeah. I tried for it to create a SVG with scissors and it was hopelessly overwhelmed. I think at least the SVG design niche will be safe a little while longer

I think it's beyond decent. I don't understand how people are not more impressed by this. Just a few years ago the only expectation would be garbled nonsense.

> the bicycle frame is the correct shape

No, the handlebar is wrong. The handle bar is rotating the frame instead of rotating the front wheel. The handle bar should be mounted on the same line as the front wheel is.

Hopefully 4.9 will read my comments :)

Could be an extremely high angle stem that just happens to match the downtube angle.

Maybe the pelican is just riding a road bike/gravel bike

I bet someone shares this link every time you post about bicycles, but since I didn't see anyone share it yet in this thread, I'll take the opportunity to do so:

https://www.gianlucagimini.it/portfolio-item/velocipedia/

Turns out even humans can be pretty bad at drawing bicycles :)

But if you need to draw a bicycle, you wouldn’t pick a random person in the street. You would hire an artist and you’d be guaranteed to have at least a believable one if not a perfect rendering.

No guarantees is why LLM is akin to gambling. Every new context is essentially picking someone out of the crowd.

On a new model release, you can guarantee two things are in the replies to Simon. One is your link, the other is "surely the models are being trained on this now"

> The most unintelligible drawing has also the most unintelligible handwriting. It was made by a doctor.

Haha

Sure, but no one is trying to force art from most people into about every area in the economy where anyone ever pays for something visual. If you asked professional artists to draw a realistic bicycle, I'm guessing few of them would try to just randomly guess what the mechanical parts looked like

As an aside, some of the renders have only a single side connection to the wheel and that is a valid bike design, the Cannondale Lefty front fork only has a left leg:

https://duckduckgo.com/?q=cannondale+lefty&iar=images&t=ffab

Glad to see that the "high thinking" level adds a helmet. Always a smart choice.

And yet some people doubt Anthropic's commitment to AI safety

Here's pelicans in all of the thinking levels - low, medium, high, xhigh, max

https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

Is the output on the max level meant to be missing?

I just fixed that (force refresh). It hit my default 8,000 output token limit, it worked when I bumped that up.

For max I used 25 input, 17,167 output which cost me 43 cents! https://www.llm-prices.com/#it=25&ot=17167&ic=5&oc=25&sel=cl...

I like the way the max pelican has a stern look on his face

low: yolo

medium: redesign bike so peli can reach bars

high: redesign bike so peli can rest on frame

xhigh: yolo

max: big peli reach bars

Max seems to me to be notably better than the others.

You've peed in the pool Simon, this has to be a part of the internal evals by now! You got to try something new - maybe a panda in a canoe?

If these were in the internal evals then the output would be much better. The 4.8 pelicans are pretty meh

Click the link

I really like that thinking level high gave the pelican a helmet.

Hey simonw I love your test, do you think using thinking level "max" makes sense for this test? I would love to see the results about it.

I don't think the API supports "max" as an option, that might just be a Claude Code harness thing.

UPDATE: My mistake, the API does support max. I added a max one at the bottom of this page (cost 43 cents): https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

The legend.

Simon, is your pelican test really captures differences among models or should you at least try like 10 times or something to average the random effects

I've been meaning to do a "run 3 times and pick the best" version for quite a while, I should really pull the trigger on that one. Currently it's one-shot only.

Best-of-3 would be cheating, ruin the test, middle of 3 makes more sense

Why would you need the 3rd run if you pick the "one in the middle"?

Middle as in not the best, and not the worst. As opposed to the second generated in sequence.

But not the best/not the worst is somewhat subjective.. so not sure how well that would work.

I think GP meant picking the median pelican

You could run 3 times and overlay/average the images to show how consistent they are

The vast majority (if not all) of these make it impossible to turn, among other fun things. Only out of curiosity, have you tried prompting further with how a bike must operate to see if it does the right thing?

tried it myself, not much of difference

https://gist.github.com/fendy3002/3026a8c4d67d1301666ec40fc0...

looks like the model already trained well on both bicycle and pelicans

Sadly I think the correlation between this benchmark and performance is starting to break down imo. Still a legendary idea will be remembered and ingrained in the models forever haha

That little red hat on hard mode is sending me. 4.8 has whimsy

I find the most miraculous thing about 4.7 to be that the pelican is facing left, wonder why the right facing everything is so ubiquitous in these images.

It's facing left but looking right...

Profound political commentary?

This happened to me in elementary school. We were doing fingerpaintings using plasticine. After all the bikes were hung on the wall, mine was racing the other way... Somehow it really stuck with me.

What do you think it means?

Eventually the frontier model folks are going to pick up on your pelican on a bike test and bake-in flawless results for that particular request.

I actually like the 4.7 the most, interestingly enough. Not like you can "objectively" weight artistic output like this.

It's funny that we've reached the level where LLMs draw more correct bikes than any random person

I don't see how a frame without a headtube can be "the correct shape".

For comparison, what's GPT-5.5 producing today?

The reasoning xhigh one is pretty solid: https://simonwillison.net/2026/Apr/23/gpt-5-5/#and-some-peli...

Lends credence to my vibe-based assertion that GPT-5.5 > Opus 4.7 (and now 4.8), which is why I've cancelled my Claude plan. Opus 4.8 is them seeing it reflected in their own numbers and having to pull stopgap measures to avoid falling behind while they embargo Mythos.

thanks for always providing this very much on time. I'm wondering what the next, harder challenge could be? Maybe some animated svg?

Is the "opossum riding an e-scooter" benchmark in the works for Opus 4.8? ;)

Good call, it's cute: https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304... - but nothing like GLM-5.1: shttps://static.simonwillison.net/static/2026/glm-possum-esco...

Am I allowed to say that pelican's little helmet is adorable? I can't provide a strong computational proof, or even a shred of anecdata...

...but that pelican's little helmet is adorable.

4.7 reigns supreme IMO.

Reply on news.ycombinator.com