I needed to thoroughly test rerankers on my companies rather unique corpus.
Opus and I wrote a parallelized test harness and labeled groundtruth in around 2 hours.
In 2022 that would've likely been all I did for a couple sprints
Opus and I wrote a parallelized test harness and labeled groundtruth in around 2 hours.
In 2022 that would've likely been all I did for a couple sprints
That sense that you did something better in a few days than you would have in a month 5 years ago. It's like buying a table saw for wood working.
One crazy thing I think about often is how there are so many correctness and testing harnesses that would have taken weeks to build in the past so we simply never would have. We'd just do our best then wait and see what comes to the surface. This is a huge part of what makes it possible to actually make better software with LLMs in my opinion. It isn't just 'LLM codes better than I ever could' (that's often untrue still) but 'LLM enables me to make assertions about the program to degrees that would have been absurdly impractical in the past'. It's huge
In this case the design was also AI generated, and there were limited wins to be found because the design was already superb.