- Where do you store your prompts? - Do you use version control? - How do you test prompts after editing? - Where do you store your test sets? - Do you evaluate results? If so, how? - Are you fine-tuning models like GPT-3.5 for better/cheaper results?
Most of the score dimensions are deterministic but we've added some where we integrated an LLM to do the scoring (... which brings the new problem of scoring the scoring prompt!). We also do a manual scan of the outputs to sanity check. Not doing any fine tuning yet as we're getting pretty good results with just prompting.
and what about llm-scoring: does LLM output passed/not_passed or it is more?
For some of our deterministic LLM tests, we have inputs that have known spelling errors but no wrong word errors, or some other combination of errors. If the config under test doesn't identify the issue, or identifies issues that we know aren't there then it's marked as being wrong for that test case. Then we can test across config x language x kind_of_error.
For the LLM vibe driven scoring we have it set up to just do a head to head between the current leading config (usually what's in prod) and the new candidate config rather than generating an abstract score. It will flag "x config straight up failed question N based on some_reason(s)" so that we can manually check it.
My partner wrote the testing framework. She's been thinking about cleaning it up and open sourcing it.
Please share your current status :)