I just found out about it and the first impression I got was this is basically a LiteLLM wrapper, a gateway/translation layer for OpenAI requests and a dashboard.
Is this an oversimplification of what that product is or not really?
To me, it feels like they just needed an "AI" product to say that ngrok also does AI stuff too!
"LiteLLM plus a dashboard" is a reasonable thought. Especially since gateways generally share a lot of similar plumbing, and I think LiteLLM is a good open source option. If you only need a translation layer and are happy to operate one on your own I think that LiteLLM would be a good fit for you.
But, ngrok.ai is for teams that want to get started quickly without having to setup and scale anything themselves. I can tell you for certain it is not LiteLLM underneath the hood and relies on the same global networking infrastructure as ngrok.
Since it leverages the ngrok infrastructure, it can route to models running on your own infrastructure in addition to public models and comes baked in with per-key access controls, automatic failover, and usage visibility. How you want to scope your keys is up to you since they can represent people, applications, or environments.
We've definitely put thought into the primitives, and while it's pretty bare bones right now, we're continously improving it every day.
I'm always open to hearing about your needs, what you think we are missing, and how we could improve.
Feel free to reply here or message me at niji@ngrok.com