Here is our take: None of these solutions are built to fit seamlessly with the data team's workflow to collaborate and work together on (imagine the difference of Modal and WandB had on data / ML teams). We have built and consulted on a number of LLM projects and found that prototypes were super easy to make. Iteration was incredibly difficult. Our goal was to make iteration as easy as writing a unit test.

A number of these projects are missing most Python developer's workflow. For example - we made it compatible with Pytest on day 1 because that's the most common testing suite.

Many competitor tools all make a good pass at these but our roadmap is significantly different and not catered for normal web developers etc. Our focus on building this is to ensure the first step makes sense - that we are super intuitive to add to a data team's Python workflow. (Just 1 assert statement to write a unit test).

cool. thanks for answering!

I'd add ours too, although we're trying to be an end-to-end one-stop platform.

https://github.com/agenta-ai/agenta

you and everyone else, unfortunately. gonna be brutal out there and take exceptional entrepreneurship to be the actual end to end platform that wins

This is a fair question on the surface, but one thing that occurs to me — given the sheer number of similar-looking things people put out, people may not have the time to exhaustively compare their offering with others. You often only discover differentiation by diving deep into each framework/lib etc, and if you dive into every Twitter-amplified new framework/package then you would never have time build your own thing.

So I guess what is a builder to do? Probably focus on practical use cases, show adoption and let these speak for themselves?

for what its worth, i agree, i've had this exact chat with @areibman (who is replying downthread). this is why i asked for market insight rather than a competitor by competitor diff.

Got it, that makes sense

Are there any large or publically shared evaluation datasets and corresponding question and answer validation data?

A few publically shared evaluation datasets include:

- HuggingFace Evaluate (and their 3 dashboards) - OpenAIEval - AnthropicEval

I would not use those for production.

I would assume your backend model is already tested on these.

You would need a novel data set.

Thanks for the enormous amount of interest and questions in this post. I wanted to make a follow-up comment to answer all the questions above about differentiation, where we see issues in the current solution space and clarify the problem we are trying to tackle.

Starting with the problem - the problem we are trying to tackle is to make iteration of LLMs and Agent applications as easy as possible for data teams - not for people to quickly edit and compare prompts (although that is 1 way of doing it). There are a number of solutions out there to help you test prompts etc. but many fail to fit within the data team's workflow (which consists of tooling like Pytest and CLI-first approaches) and do not make it easy to iterate and launch on things like agents.

From conducting 30+ interviews with ML Engineers and data scientists building in this space - all of them want to first build a LangChain agent/RAG pipeline and then build their own internal version of it (due to the fact that LangChain is quick to set up but lacks tooling). A lot of them are encountering issues with developing the right evaluation infrastructure (which DeepEval aims to solve through synthetic data creation and easy-to-use testing tools).

Our product roadmap is to not only build the initial unit testing for LLM but also to make it easy for other developers and other MLEs to quickly iterate off this. This means long-term, our plan is to ensure that our users and customers are able to build the best agents/LLM solutions possible.

And (to be frank) a lot of the existing solutions aren't the best looking, have limited visualisations (or are just dead)

Sorry for the n00b question here, but how would you prevent the next version of the foundation model from spidering and learning from the exam questions themselves, and thus "cheating" on the exam?

Thanks for the interesting question!

From what I've observed from both academia and posts from a number of top organizations (with seemingly the exception of OpenAI) - all the models are trained on test data. That's why OpenAI seems to report poorer results than the rest but perform the best in practice - they seem to have their own curated dataset.

I have arguably the opposite opinion here - I don't think we can "prevent" cheating but we should instead put in the work to create new exam questions and make it easier to do so. I feel "learning" the exam questions is (IMO) completely fine and should be expected (why should we have a static dataset if it diminishes the quality of the results) - much like having new exam papers every year for students.

What I expect going forward is for the next version of foundation models, there will also be smarter ways to synthetically create datasets from ground truths themselves (we made our first pass at it here - https://docs.confident-ai.com/docs/quickstart/synthetic-data... ).

I feel that data labelling companies will hold an edge and that the cycle of finding new edge cases, labelling, continue training will continue in the foreseeable future (see Karpathy's data engine flywheel in 2021/2022 Tesla AI day).

Microsoft Guidance integration?

https://github.com/guidance-ai/guidance

Interesting - I'm not really sure how this would work but I'm happy to discuss this in more detail - I see this being as a suggestion for what to do when tests fail. Happy to discuss this via email if you'd like.

Feel free to email me at jacky@twilix.io

I might be missing something. But I was thinking so that I could test a chain, rather than just a single prompt.

Got it - that wasn't immediately intuitive from the suggested comment. Yes! Our roadmap will include ways to test other parts of the LLM application stack very soon!

The docs for this aren't great - leaked api keys, code doesn't run, etc...

Hi thanks for letting us know - please feel free to let us know of any more bugs - we'll make sure to write better docs and fix the code. We're just a few days in!

I logged them as issues for you, good luck! As a test engineer it's great to see more test tooling around generative AI :)

Resolved all issues - thank you!

How is this standing out from https://benchllm.com/?

I really dislike benchllm's use of yamls for test cases - I'd rather it be in code.

""" input: What's 1+1? Be very terse, only numeric output expected: - 2 - 2.0 """

Agreed. No one should ever have to touch YAML for writing unit tests for LLMs. Ever. Most people writing agents and LLM applications are Python developers/data scientists/ML engineers.

Reply on news.ycombinator.com