- https://github.com/BerriAI/bettertest https://twitter.com/ishaan_jaff/status/1665105582804832258
- https://github.com/AgentOps-AI/agentops
- https://www.ycombinator.com/launches/JFc-baserun-ai-ship-llm...
- https://news.ycombinator.com/item?id=36958175
- https://github.com/promptfoo/promptfoo
can you articulate what insight on the market you have that will make you stand out over time?
So I guess what is a builder to do? Probably focus on practical use cases, show adoption and let these speak for themselves?
Here is our take: None of these solutions are built to fit seamlessly with the data team's workflow to collaborate and work together on (imagine the difference of Modal and WandB had on data / ML teams). We have built and consulted on a number of LLM projects and found that prototypes were super easy to make. Iteration was incredibly difficult. Our goal was to make iteration as easy as writing a unit test.
A number of these projects are missing most Python developer's workflow. For example - we made it compatible with Pytest on day 1 because that's the most common testing suite.
Many competitor tools all make a good pass at these but our roadmap is significantly different and not catered for normal web developers etc. Our focus on building this is to ensure the first step makes sense - that we are super intuitive to add to a data team's Python workflow. (Just 1 assert statement to write a unit test).
- https://arxiv.org/abs/2308.03688
(author of agentops here)
- HuggingFace Evaluate (and their 3 dashboards) - OpenAIEval - AnthropicEval
I would assume your backend model is already tested on these.
You would need a novel data set.