Hacker News Reader

Comment by boredumb | Hacker News Reader

I'm integrating llama 2 7b with an application I'm building out currently and one of the biggest reasons was privacy, followed closely by price and lastly by getting it to work locally in a few minutes.

I built a now abandoned project using the GPT API and it was fine and not terribly expensive for my use case but customers didn't like the pay for usage model and the alternative was do weird UX to limit people abusing the prompts into something I couldn't afford bootstrapping as a side project.

View on HN

Can you elaborate on the pricing difference?

OpenAI costs money and llama 2 i'm able to run on my GPU so at least for development purposes at the moment it's ""free"" for me to experiment with - mind you this is a side project with zero funding outside of myself, ymmv if you have access to funding to a point of making OpenAI tokens totally disposable.

Ah okay, yeah I was curious how much you'd pay running it on a cloud server in a production-type scenario. Thanks

I have a vultr account I just logged into and checked, I could rent enough GPU memory to run llama 2 for ~180$ a month from my very quick look - so if I make more than 27000 requests with a 4k token payload I will break even - otherwise I could be using openAIs API and be making out better.

EDIT** im sorry that was gpt3.5 - with gpt4 it would be 1000 requests before I broke even at 180$

LLAMA 2 appears to require no cost because it can be run locally. The license does mention that if you are using facilitating than 700 million users that you have to negotiate for a different license or something like that but for most peoples uses, it would seem that LLAMA 2 is basically "free".

Thanks for elaborating. Yeah I was curious how much you'd pay running it on a cloud server in a production-type scenario. Thanks

Well it's free as in free hops. You still gotta buy/rent the brewery to make the free beer.

Exactly. It isn't free as in FSF definitional freedom, you still can't do certain things with it etc but it is unencumbered by external costs unless you exceed a certain usage threshold.

Right, but can you scale GPUs for cheaper than OpenAI charges to use their APIs? 3.5 is _cheap_, and perfectly good for many use cases.

The long-term price difference can be hard to estimate. Suppose that your application is heavily dependent on GPT-4. OpenAI can double your API prices the next term. Or they can decide to stop supporting the model that you wrote all your custom prompts for. Or they could decide to disable your account because they feel like it.

Unless it's for some fringe feature, building your business on OpenAI is probably a considerable (financial) risk in the future.

These are definitely very significant risks. But some of those risks are hard to avoid unless you train your own model, which can be prohibitively expensive.

Say you're building on top of llama and Facebook decides not to update it any more or change the licensing terms (again). Say you're building on some other "open source" model and that project dies.

At least you can keep running the existing model rather than getting locked out over night. That's definitely much better. At least it's survivable. But you would still have to find alternatives and review/scrap all your custom prompts.

Reply on news.ycombinator.com