Hacker News Reader

Comment by adaboese | Hacker News Reader

I cannot really wrap my head around what is this. Is this the whole model? Is this some sort of middleware between user and OpenAI?

View on HN

It's someone taking llama.cpp, strapping it to a HTTP server lib and implementing the OpenAI REST API, then putting some effort into a shiny website and docs, because AI-anything is a reputation and/or investment magnet right now.

Not that it isn't a nice idea -- making it easy to test existing OpenAI-based apps against competing open models is a pretty good thing.

Or… it’s just someone who wanted to build cool stuff.

I see the value in being able to locally run apps designed to talk to OpenAI apis.

There are dozens of other endpoints that will do this, and possibly do it better.

I'm all for cool innovations, but... I'm not really seeing the advantage here beyond the shiny website to attract VCs.

"Nitro is a drop-in replacement for OpenAI's REST API"

It's not a model, it puts an API on local models.

so I still need to download and host models myself.

I found that to be incredibly impractical. I tried to do it for my project AIMD, but the cost and quality just made absolutely no sense even with the top models.

Well, the market for local inference is already quite large, to say the least. “It didn’t pencil out in my business favor” doesn’t seem like a fair criticism, especially for an app clearly focused on the hobbyist—>SMB market where compute costs are dwarfed by the costs of wages and increased mental load.

I definitely see your specific point tho, and have found the same for high-level usecases. Local models become really useful when you need smaller models for ensemble systems, to give one class of use case you might want to try out —- e.g. proofreading, simple summarization, tone detection, etc.

Our you could use something like omnitool (https://github.com/omnitool-ai/omnitool) and interface with both cloud and local AI, not limited to llms.

Reply on news.ycombinator.com