Hacker News Reader

So basically "one-shot <complicated thing> using <x> testing methodology".

I don't think that's a very effective test of an agents ability.

It's intended as a "test to see how effective agents are when guided by someone with no expertise in testing who's maybe heard that you should apply certain techniques or use certain libraries." What do you think would be a more effective test for that purpose?

So not "How well do agents use test/verification techniques?" like the title says then.

It just looks like a lot of effort to go to in a way that doesn't answer the question the title poses.

I'd love to see something that could guide how to set an agent up for success rather than this naive approach which only really answers the questions that most of the audience here probably don't have.

Reply on news.ycombinator.com