Hacker News Reader

Comment by simonw | Hacker News Reader

Pelicans. Sonnet 5.5 has the same problem as Opus 5.5: on "max" thinking effort it burned through 128,000 thinking tokens (taking 15 minutes to do that) and ran out before it had produced the final SVG.

https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

Here's how the thinking effort levels compare:

  low
  27 input, 1,623 output, thinking_tokens: 0
  1.6284
  Duration: 10138ms (10s)
  
  medium
  27 input, 1,796 output, thinking_tokens: 0
  1.7914 cents
  Duration: 11266ms (11s)

  high
  27 input, 2,334 output, thinking_tokens: 745
  2.3394 cents
  Duration: 17376ms (17s)

  xhigh
  27 input, 5,730 output, thinking_tokens: 2535
  5.7354 cents
  Duration: 41882ms (41s)

  max (failed to return response)
  27 input, 128,000 output, thinking_tokens: 128000
  $1.28
  Duration: 940617ms (15m 40s)

Low and medium both used 0 thinking tokens.

View on HN

Does anyone really still care about these pelicans?

Any model release it’s the top comment, I do not understand why.

Karma farming by parent commenter and HNers’ tendency to upvote low quality content (not dissimilar to other social media networks)

Is this low quality content relative to most HN comments?

Very few HN comments are high quality

Mainly because they're funny, but it's also because I try pretty hard to make the comment more interesting than just "here's a pelican". In this case I used the pelicans to talk about the 128,000 token limit bug at "max" and share comparative pricing.

In the GPT-6 comment I included full visual comparison grids: https://news.ycombinator.com/item?id=49805509#49806126

For DeepSeek v4.1 Flash I identified that the OpenRouter reasoning levels are mapped to a smaller set of levels for that model: https://news.ycombinator.com/item?id=49639090#49645591

Agreed. It was a creative and unique test for a while. Now, no offense to the author, it feels like every conversation about a new model is dominated by the pelican on a bike posts as they always become the top comment.

You can click the little [-] icon next to the post to collapse the entire sub-thread. I do that all the time.

It's an easy way to compare the coding and creative strengths of models. I prefer them over reading a tabular comparison of benchmarks which you have no real insights into.

Because hn has some kind of a community and not every comment is gold (see yours for example) and people are able to skip comments if they don't enjoy them?

I can't understand it. Clearly someone cares because like you say the comments are always upvoted. But why anyone cares I simply don't know. It just feels like attention seeking behaviour to continue posting it.

Isn't posting any comment on a forum like Hacker News "attention seeking behavior"?

Seems like quite a different level of attention seeking by posting the exact same prompt result, every time there’s a new model, that links to their own website.

Linking to https://tools.simonwillison.net/markdown-svg-renderer?url=ht... should be pretty inoffensive (I started habitually linking to that after people kept complaining about linking to my blog) - that page renders Markdown with SVG embedded in it, but doesn't link to the rest of my site at all.

This is a community and it is an inside joke at this point, it wouldn't be a proper model release without Simon's pelicans.

I find it useful (as well as a fun art project).

Sick of the cheap shot

I would not say I care, but I do have curiosity. I expect one day they will start adding some textures or something like that.

It would have taken me a while to stumble upon this "running out of tokens" on MAX thinking issue without his trials and post. I've seen the pelicans for years now, and if they stopped coming for some reason, I would probably go to his site to catch up on recent models and findings. So I don't mind them

I do, it gives some fun comparison between models.

You can also check for any kind of degradation of them - you have the prompt, it doesn't use much $.

This is evidence that Sonnet 5.5 wasn't yet trained on the HN comments from the Opus 5.5 release. Maybe Pelicanmaxing will lead to 127000 thinking tokens being used on Max.

If it was trained on HN, there would be a 60% chance of it just saying "I'm so tired of this request, can we please move on"

I'd say:

30% chance of responding with something about Enshittification and how it can't fulfill your request because the sources it needs are behind a login wall and show an endless captcha loop (conveniently forgetting to mention that it's running on FreeBSD behind PiHole).

30% chance of complaining that it's being subsidized and that "prices are going to go up bro."

30% chance of some unrelated rant on ID checks for age verification.

10% chance of a different rant, this time on how nobody took Snowden seriously and how terrible Flock is.

The contrast between Anthropic, who seem to be training their models to output ever-increasing numbers of reasoning tokens, and Fireworks's Ember-1, which was explicitly trained to preserve the quality of a model's responses while cutting down on reasoning, is interesting. Claude Code also uses more many tokens per task per model than any other harness in benchmarks.

Anthropic's "Max" modes seem like a yolo mode: "use 10x the tokens to try to break the hardest possible problems". But their models don't seem less efficient at normal reasoning modes.

I can't see Ember on AA's index yet, but their post claims "half the reasoning tokens for the same answers" as Kimi K3.

That would make it about so, I assume?

             Score  Tokens  Reason  Cost 
 Kimi K3 Max    44  48k     32k     $2.00
 Half reason    44  32k ?   16k ?    ?

 Opus Med       51  26k     12k     $1.34
 Opus High      54  36k     18k     $1.82
 Opus Max       58  119k    84k     $5.98

 Sonnet Med     41  ?       ?       $0.59
 Sonnet High    47  ?       ?       $1.08
 Sonnet Max     56  193k    142k    $7.60

Medium is Anthropic's default.

Having a less efficient mode isn't necessarily a mistake -- the purpose of configurable effort levels after all is to be able to put more thought into a problem.

This is great news because it means the model has not been benchmaxxed on stupid metrics.

PS: the next human that brings up pelicans on bicycles should try to draw them.

I like the one where the pelican is using the non-pedalling leg to control the handlebars because its wings won’t reach!

I feel the fact that these models always modify the body design of a pelican to fit the bike rather than the other way around represents a fundamental issue with AI.

Where do you run sonnet/opus where you are limited to 128k, given they are both 1M context window models?

That's max output tokens per response limit, separate from context length

It's the output token limit, which has been 128,000 for Claude models for quite a while note

Pretty crazy that the model doesn't know that it needs to stop before it hits 128k output tokens. I guess it has no sense of how many tokens in it is? Wouldn't this be possible to work into the architecture?

I think this is a bug. I've not seen this problem from any of the other frontier models.

Do other models put a hard cap on the output tokens it can generate?

Yes, the OpenAI GPT-6 Astra limit is 128,000 as well: https://developers.openai.com/api/docs/models/gpt-6-astra

Gemini 3.8 Flash is 65,536 https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flas...

I would also consider this a bug. I think ajy reasonable consumer would.

Sonnet 5 had the same problem with ‘max’. In a free sub, I would never get an answer back even for very simple prompts. It would just churn on nothing and return max token usage reached.

I’m not sure whether that’s a feature or a bug at this point though.

Thank you for the pelicans sir, how do you think they compare to other models in Sonnet’s pricing/capability range?

what's most surprising is the difference between high and xhigh

max thinking means benchmaxxing i guess

that pelican one-pedaling

“Pelicans are solved.”

I think the next models will be benchmaxxing on the Pelican benchmark tbh

wow nobody but you has ever thought of this and certainly simonw has never addressed this

Reply on news.ycombinator.com