I wrote 2000 LLM test cases so you don't have to: LLM feature compatibility grid

Those tools map API compatibility. These tests+config add:

1) check which features are available

2) check which parameters you need to use for best results. For example, there are about 6 different options for requesting JSON from OpenRouter, and different models work best with different options.

3) check that the features consistently work. API compatibility and functionality are not the same.

4) Go much deeper: are the models good enough for synthetic data generation? Can they generate uncensored model inputs if you're building a toxicity eval? etc.