Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by Demotooodo | Hacker News Reader

Full thread

Demotooodo·

Price.

Data privacy.

Controlled latency.

Plenty of reasons to not send arbitrary data to a third party service.

View on HN

There's also the availability factor. OpenAI has been known to go down on occasion and without warning. If a product relies on an LLM, I wouldn't feel great about the observed uptime of OpenAI APIs.

Another side of availability is that they'll make changes to the model without warning, which alters the results of the prompts you already have written. Developing against their API is developing against a moving target.

FWIW, OpenAI's availability seems to have gotten significantly better since May when we launched with them. I monitor our availability Service Level Objective and we keep needing to increase the success rate because they keep improving things.

This doesn't take away from high availability being a legitimate need to host your own LLM, though.

Can you elaborate on the pricing difference?

Free vs monthly cost. What is there to elaborate on?

It’s gona have to be hosted and run from somewhere…

Hosting your own LLM is anything but free. Aside from the constant operational expense with people monitoring and fixing issues, you need to provision enough resources and run your own inference server, which is both nontrivial and likely to perform far worse than OpenAI. There's legitimate reasons to host an LLM yourself, but it's not a "make this cheaper" button.

There may be a tipping point where you're burning XXM/year in API costs and the maintenance cost of rolling your own can be justified.

In the short term I agree, and one thing to consider is how rapidly the space is evolving and whether your team can even keep up with the latest advancements.

However, there will come a time when the bill comes due after launch and it will be very tempting to hire people to reduce the CapEx on the API.

Reply on news.ycombinator.com