Not exactly related, but it makes me wonder what kind of metrics you can devise for tracking "regressions" on LLMs.

I feel like OpenAI definitely has thought about this at length, but I'm curious about what matters most for them / raises OpsGenie/whatever alerts. Internal model metrics? Customer usage patterns? Random test conversation diffs?