I have seen this, but I find it fairly hard to use.
Our goal is to focus on datasets and make it very easy to create and manage data.
In our next release, we will be launching a way to do this using a UI.
30 karma · joined June 29, 2022
Our goal is to focus on datasets and make it very easy to create and manage data.
In our next release, we will be launching a way to do this using a UI.
Apart from this, we generate a diverse set of questions including complex reasoning and chain of thought.
We also generate domain specific unsafe questions - questions that violate TnC of the particular LLM to test the model guardrails.
We are working on getting SOC2 certified. In the meantime, we sign a legally binding agreement with our users who have data privacy needs/concerns.
We incorporate user prompts to generate the outputs and provide diagnostics and feedback for improvement, rather than eval metrics. So you can plug your low scored queries provided by Ragas, your prompt and context. FiddleCube can provide the root cause and the ideal response.
This is an alternative to manual auditing and testing, where an auditor works on curating the ideal dataset.
On a side note - I love how quickly your team is shipping! Do keep it going!