Comment by lmeyerov | Hacker News Reader

Comment by lmeyerov | Hacker News Reader

We've been having to build this inhouse for louie.ai to be more engineer-grade as all the libs we evaluated weren't flexible enough for common scenarios, so maybe a useful evaluation laundry list for makers & users here:

* 12-factor: support setting via env vars, config files, api, etc

* decoupling auth config from model config

* supporting registration of multiple auths & models, not just one, including via 12-factor

* streamlining ability of llm apps to negotitate which llm models

* inferring & validating matchup of what your model provider gives and your app configures & requests, ideally at config or load time, and in a testable way

* transparent native support for each provider, as each provider & api has annoying deviations & useful features that end up being relevant. Ex: even openai vs azure openai has differences like the notion of 'deployments' and around rate limits that should be handled but also exposed

* observability: introspection hooks, including configuration for opentelemetry metrics, telemetry, & logs, including tenant/user dictionaries

Without that kind of stuff, a third-party dependency is more annoying than useful for 'serious' implementations, b/c we ended up fighting the library vs using

(And we'd be happy to OSS etc if relevant.. such a bear!)

> weren't flexible enough for common scenarios

Does openAI support that stuff, or is that a more general MLOps (or whatever you'd call it) list of functionality you need? The API call aspect seems like a pretty easy part that's not a barrier to switching models. It's whatever customizations for a given application that I'd expect to be more work. But re your text I quoted, I'm always wary of "one line of code" solutions that are harder to customize than it would be to just write the the thing.

openai and langchain natively support all of that or expose the hooks to do it

all that stuff is what we needed to do to support multiple providers vs hard-coding a single one. none of it is specific to our codebase. if we were going to take on a third-party dependency, we'd need it to be serious about this kind of thing, else we have an external dependency to work around on our critical path

Edit: As an example, if the switching layer doesn't implement model negotiation and doesn't expose key model details (which vary by model provider service, and often require REST calls to introspect), we can't add model negotiation / retries / etc on top, and would have to edit the innards to enable that, which opens up all sorts of questions

What is “model negotiation” in this context?

Ex: An agent pipeline is asked to generate some code, triggers a code agent, and azure openai is hooked up with both primary and fallback service regions + a few deployments of 3.5 turbo, 4, etc. Which model gets used first, second, ...? Why? And for the model abstraction layer, what does it need to expose vs hide?

very relevant. OSS please!

Reply on news.ycombinator.com