Hacker News Reader

Comment by prymitive | Hacker News Reader

I often find myself annoyed when Opus fixes a typo in a comment and decides to run tests, lints and whenever else it can find to run. Often it will start by stashing current changes just to preemptively check if all tests were passing before. And I can blame myself a bit because my rules do say: verify all changes with tests. But as there is that I in AI that is hyped which you’d think means it knows not to put tomatoes into fruit salad …

View on HN

> [..] my rules do say: verify all changes with tests

I am a bit surprised that you're disappointed that it does exactly what you told it to - people usually have the opposite complaint.

If you're using it interactively and watching what it changes, I'd trigger the tests when you think it's needed. And if you want to go more hands-off, why not add try to encode the same nuance you'd use into the rule?

Rather than bake that into the prompt - wouldn’t it be better to just set up a pre commit hook that runs tests and linting?

Maybe, depends on their workflow. In my human workflow, I tend to use commits as checkpoints and then squash before pushing. I'd usually only run time-consuming tests before squash+push.

But yes, anything you want to ensure really needs to be a hook.

edit: realizing with "precommit" you probably meant a git hook not one in their harness. I'd have written the same response more or less though. :)

Oh yes - definitely the git kind of hook. Also, I always forget that there’s a pre-push hook as well. So you don’t need to do things every commit.

But then you could just be storing up a lot of problems…

Indeed. That's why I think it depends on the individual's workflow where it should live.

Following rules like "verify all changes with tests" down to a tee is usually a desirable trait in LLMs. Personally I'd leave that behavior there (just like with humans for some tasks like aviation you have them go through checklists even if some stuff you can infer is not needed). But otherwise just make it "always run tests unless you're absolutely sure they can be skipped".

Add "... unless the changes are trivial, docs-only, or typo fixes" to the "always verify with tests" instruction and see how that does

That's one of the reasons I started https://beolis.com. Now I have a workflow that says to do things in a TDD and run only new tests and related tests but NEVER run the full suite and then when it finishes the workflow runs all the tests -- if it works, great, if it doesn't then the workflow continues, feeds the output to a cheaper LLM to summary the errors and then get another run to actually fix it, based on the failures and the context based on what should be implemented.

-- note: I've been full time in Beolis for some months already, feedback welcome ;)

Reply on news.ycombinator.com