Hacker News Reader

Comment by ramigb | Hacker News Reader

Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe? From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly use claude (opus) on typescript react projects that are well setup with minimal plugins/MCPs!

happy to share more if you are interested.

View on HN

The reasons are highly project specific. The closer your project is to CRUD, the higher the chance of success.

yes ... 'the closer it is to normative, the higher the chances of success'.

> For instance the task naming in the task file starts with an optimistic 1, 2, 3, 5, 5a but then eventually gets to 8a, 8a1, and then ends up with 8b2c2b3 and “8b2c2b2b checkpoint1”. The code that it produced got ever more wild. I don’t want to bore you with what it tried to build, but here are some example pieces of the interpreter changes:

  Hardcoded constants everywhere
  Multiple same-line macro invocations in C
  Random indexes in production code
  Hideous tokenizer code in C

https://lucumr.pocoo.org/2026/9/7/astra-why/

Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation.

It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution.

In fact, that was fairly counterproductive since this codebase dumps events to the topic per company so you'd get a lumpy distribution where only one partition is ever active at once (starving other consumers)

The fact it decided it should add this without prompting was a bit shocking and I'll probably try to make it explicitly clear in the context not to "do random shit that wasn't asked for"

One thing I've always been curious about is, often times it seems that models don't seem to have these addendums like "dont do excessive/random shit" by default? Or I suppose, if it did have something like that, and still screws up like in your example, it clearly isn't working, so assume that it doesn't.

But the phrase "don't do random shit", semantically, from a monkey's paw perspective, could imply curtailing the model's creativity and 'thinking out of the box' capacity, that might have existed in its 'reasoning' process. So I'm always concerned if it's possible that, adding these phrases might be part of the reason why a model performs dumber than it should.

I don't have empirical evidence to support that supposition though.

I think of it as a tradeoff between creativity and specificity. Every instruction you give reduces creativity, and at best, increases specificity (I imagine a lot of prompts like "make no mistakes" do literally nothing but pollute context, but I haven't evaled them)

So if you're doing something very ordinary, fewer instructions result in better results. If you're doing something fairly off-piste, you have to give instructions to that effect and accept less creativity. For situations where you want it to do something extremely specific, tons of instructions and accept that you're going to get much closer output but much worse "intelligence"

Another way to think about it is Type 1 and Type 2 errors or sensitivity and specificity from statistical testing - do you want an agent that solves any problem but goes off the rails 10% of the time, or do you want an agent that can only solve 10% of the problems but nails them 100% of the time (sensitivity and specificity, respectively)

>could imply curtailing the model's creativity and 'thinking out of the box' capacity

Imo a feature. I can handle the creativity and out-of-box thinking just fine. I want the model to do the grunt work but be smart enough to do the grunt work correctly.

This is why I like to really interrogate Claude on what it’s planning to do, what the implementation will look like etc before writing code. Without enough direction and time thinking about it, it can definitely go in an unintended direction

I usually use Fable with superpowers to generate plans and do subagent driven development but unfortunately I was at my Fable weekly limit with some other usage left.

I forgot how awful using Opus 5 directly is.

If it works but the design is terrible - is that still success?

In the short term yes. Long term, no. But I guess you can say the same thing about a team of engineers taking shortcuts

Horses work, Telegraph works, Gas lamps work, etc.

this is a good question. what would make you reject an otherwise working PR on design grounds?

here's where all the models messed up!

its under this section 'Missed requirements are the most common failure' on realswe.withspecific.com

we also have the setup in the blog. the reason for lower success rates is that we gave models ambitious tasks that real engineers worked on for weeks.

Reply on news.ycombinator.com