Show HN: FiddleCube – Generate Q&A to test your LLM

Convert your vector embeddings into a set of questions and their ideal responses. Use this dataset to test your LLM and catch failures caused by prompt or RAG updates.

Get started in 3 lines of code:

```

pip3 install fiddlecube

```

```

from fiddlecube import FiddleCube

fc = FiddleCube(api_key="<api-key>") dataset = fc.generate( [ "The cat did not want to be petted.", "The cat was not happy with the owner's behavior.", ], 10, ) dataset

```

Generate your API key: https://dashboard.fiddlecube.ai/api-key

# Ideal QnA datasets for testing, eval and training LLMs

Testing, evaluation or training LLMs requires an ideal QnA dataset aka the golden dataset.

This dataset needs to be diverse, covering a wide range of queries with accurate responses.

Creating such a dataset takes significant manual effort.

As the prompt or RAG contexts are updated, which is nearly all the time for early applications, the dataset needs to be updated to match.

# FiddleCube generates ideal QnA from vector embeddings

- The questions cover the entire RAG knowledge corpus.

- Complex reasoning, safety alignment and 5 other question types are generated.

- Filtered for correctness, context relevance and style.

- Auto-updated with prompt and RAG updates.