Hacker News Reader

Comment by cbg0 | Hacker News Reader

Do you not have unit tests, or how does it introduce regressions? You can tell it how to run the test suite in CLAUDE.md

View on HN

Opus 5 tries to modify the unit tests as a cover to its own regressions - thinking its own logic is correct and the test must be wrongly specified

I've set permissions of the existing unit tests to read-only for this reason, since I've seen all the recent Opus do this.

Sometimes it will also not careif a unit test fails, calling it "errant" or "legacy" or something, where it then removes it from the list (not sure how to get around that one, other than a read only launcher).

I found it eagerly reversing existing product decisions like changing a user given date into created date, since it thought creating something that is in the past is incorrect. This was not even related to the task at hand at all. I noticed this kind of stuff happens more with ultracode for some reason.

Any subagents-based workflow is prone to this, because of the fragmented context(by design).

It is a well known fact that projects with unit tests never have regressions.

I don't understand the need for the snarky comment, LLMs can run the test suite and avoid regressions.

A change can introduce regressions in any large project even if 100% of unit tests pass. Unit tests test individual units, regressions can happen at many levels. Especially if we treat performance degradations as regressions.

I detect the regression already in planning with Opus 5, so I do not let Opus 5 implement anything. But it is a waste of time and tokens! Does planning with Opus 5 works out for you?

It sounds like you just need to correct the plan it lays out to avoid the regression? I'm just looking to debug with you, not defending the model. I've mostly used Opus 5 for code reviews & bugfixes.

Yeah fine. I mean my plan wasn't to difficult. For example this morning I started with Opus 5 to tackle a problem. During planning at some point Opus 5 detected _8_ regressions in it's own planning, after I directed it towards those potential regressions. So, in this very moment now, Fable 5 implements code already and the planning before with Fable, done with the same instructions, was flawless and quick. And I am sure, Opus 4.8 would be flawless either. Same harness, same claude.md, etc..

Do you think it might be overthinking? Try it on medium effort.

Probably, I ran it on xhigh. Worth a try.

Reply on news.ycombinator.com