Show HN: A lightweight AI gateway to 100+ models, in TS | Hacker News Reader

Show HN: A lightweight AI gateway to 100+ models, in TS

github.com

37 pointsroh26it13 comments

We are Rohit & Ayush, we created Portkey this year March to help tackle some challenges we had seen while building apps based on GPT3, 3.5, 4, and the DevOps principles we brought to the scene to help tackle them.

We believe, a solid, performant, and reliable gateway lays the foundation to help build the next level of LLM apps. It decreases excessive reliance on any one company and takes the focus back to building instead of spending time fixing the nitty gritties of different providers and making them work together.

Features:

  Blazing fast (9.9x faster) with a tiny footprint (~45kb installed)
  Load balance across multiple models, providers, and keys
  Fallbacks make sure your app stays resilient
  Automatic Retries with exponential fallbacks come by default
  Plug-in middleware as needed
  Battle tested over 100B tokens and millions of requests

For the folks serious about gateway, separation of concerns, TS developers.. I'd love to hear your thoughts and we're hungry for feedback!

Reach out to us at hello@portkey.ai or explore the project: https://github.com/portkey-ai/gateway

Open articleView on HN

Congrats, this looks neat, and surely great to have more TS products in the ecosystem.

One plugin or feature that I will like to see in an AI gateway: *Cache* per unique request. So if I send the same request (system, messages, temperature, etc.), I will have the option to pull if from a cache (if it was already populated) and skip the LLM generation. This is much faster and cheaper - especially during development and testing.

Thank you! We have built out the cache system -- we do both simple caching (matching the request strings 100%) and also do semantic caching (returning a cache hit for semantically similar requests). More here - https://portkey.ai/docs/product/ai-gateway-streamline-llm-in...

The caching part isn't open source yet, but part of our internal workers. Would be very cool to open source it!

Awesome! We built the simple version in-house, and hoped someone would productize it.

A few months back, we embarked on what seemed like another regular project. Little did we know, it would turn into such a game changer for LLM deployments.

Feels surreal that this gateway is already processing upwards of 3B tokens a day in such a short time.

Pretty excited to announce this! While there are some popular and awesome AI gateways out there, like litellm, bricksai - none are written in TS, and for the TS ecosystem. Looking forward to the community's feedback

While I love and use Python everyday, I do believe that JS is a lot more popular and mainstream for web development today.

Portkey is super cool, congrats Rohit and Ayush on the HN launch!

thanks! we spent very long building it out properly. Time to integrate weaviate generators now?

Congratulations Rohit, Ayush and team Portkey on the launch!

Excited to take it for a spin and contribute!

_/\_

Congrats on the launch folks! This sounds really amazing - rooting for success!

Congratulations on this launch. Looks awesome.

Congrats on the launch! Looks amazing

Reply on news.ycombinator.com