Show HN: Manage LLM providers while factoring in Cost and Speed
github.com
We built this framework for ourselves internally and decided to open source it.
It's really similar to open router but allows you to do a couple more things
1. Allows you to factor in 'speed'. If you want your API call to go as fast as possible regardless of cost you can specify that. Otherwise it'll default to the cheapest available provider 2. Allows you to pass in a 'validator' function and 'fallback' models. This way you can ensure the response you get is valid according to your own internal logic 3. Uses the OpenAPI content/role format to interact with models so you can quickly swap between them while you test.
We've found a bunch of use cases for this (https://yc-bot.lytix.co/, https://notes.lytix.co) and it's allowed us to move much faster since we don't have to think about credentials between all our providers.
Hope you enjoy it and would love to get any feedback on the project