Show HN: Litellm – Simple library to standardize OpenAI, Cohere, Azure LLM I/O | Hacker News Reader

Show HN: Litellm – Simple library to standardize OpenAI, Cohere, Azure LLM I/O

github.com

62 pointsij2317 comments

I built this library because langchain was too bloated and I needed a simple abstraction to call multiple LLM APIs. litellm has two functions - completion(), embedding()

Open articleView on HN

Great start! Are you planning to add to following:

Retries w exponential backoff, Caching, Streaming output, Function-calling support

You can use tenacity for retries and wouldn't you want to cache the request / response around the endpoint instead of the gpt call -> that's what we ended up doing.

Streaming output and function-calling support is interesting

Could you elaborate on this — “wouldn’t want to cache around the endpoint instead of GPT call”. Just want to see if I’m missing an important consideration here

You're probably calling openai from a backend server. If you're making the call because a user asked a question -> your endpoint received it -> did some processing -> called openai -> returned the response,

why would you cache the openai call instead of the endpoint that's receiving the user call?

good points, probably going to add streaming output, function calling support. As for retries tenacity does a great job already

Take a look at llm-client a similar library that also support chat, async and more llm providers https://github.com/uripeled2/llm-client-sdk

thanks for sharing, while your library looks really powerful my goal with Litellm is simplicity

This is amazing. Really needed something like this to standardize all my different AI APIs!

On a side note - I love how quickly your team is shipping! Do keep it going!

Thanks!

This is much needed. For someone looking to quickly implement, having a simple interface goes a long way.

Thanks

Very cool Ishaan!

Thank you !

>completion(..., azure=True)

Why like this?

azure uses the openai python sdk, just remaps certain components. The models are also user-named. This makes it hard to detect if a model passed in is an azure model

azure models have custom names - eg I call mine 'chat-gpt-test1', I require some flag to know if it's an azure model

Reply on news.ycombinator.com