I saw a couple YC launches like Hegel AI.
I'm personally interested in deployments in small teams or teams with a lot of freedom to pick and choose their own tooling.
Basically you can get a Docker container that will publish an Open AI API compatible end point. You can then choose the model that sits behind that API.
As deployment will be in Kuberenetes we will clusters with GPU resources to maxz out performance but we're not there yet.
- Configurable context and cases mapped to a RESTful API
- Multi-account and high throughput error handling
- DDB backed records of all requests and responses for evaluation, debugging & training
- One-click devops deploy
Has helped us deploy and maintain LLM apps into production quite easily. Let me know if you would like access to the repo.
Makes sense on the deployment thing. I like what Vellum is doing. Helicone and Portkey also let you do deployments of prompt templates through APIs.