Hacker News Reader

Comment by flangola7 | Hacker News Reader

Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT. It comes with a number of use-case conditions and gives Meta many avenues to revoke a license if they feel like it. They also have a hard cap on the number of allowed users you may have using your Llama-based product above which you must seek further Meta approval.

Furthermore, Llama remains well below GPT-3 on human rated tests such as programming, and GPT-3 is already over three years old. It is also misleading to suggest Llama 2 can be ran on consumer hardware - the smaller and quantized models can but those are even more lacking in capability. Full power Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session.

OpenAI does not have a moat, but they do have a very high wall.

View on HN

> They also have a hard cap on the number of allowed users you may have using your Llama-based product above which you must seek further Meta approval.

This is not what the Llama 2 license says [0]. There is a cap on the number of active users of products (any products, not just ones that may make use of Llama) by the company who plans to use Llama 2 as of Llama 2 release date.

Not a “future cap on number of users of your Llama2-based product”. Also, the cap is 700 million users.

> If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee's affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta

[0] https://github.com/facebookresearch/llama/blob/main/LICENSE

> Furthermore, Llama remains well below GPT-3 on human rated tests such as programming, and GPT-3 is already over three years old.

It's just not part of the entended use-case, and hasn't been trained to do so. It's almost like complaining that StableDiffusion isn't good at text generation…

> It is also misleading to suggest Llama 2 can be ran on consumer hardware - the smaller and quantized models can but those are even more lacking in capability

The biggest models will be abble to run on the CPU just fine with llama.cpp as long as you have enough (cheap) RAM. Sure it's slow, but you can run it.

> Full power Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session.

What is that “per inference session” doing here? You pay the hardware only once you know… (and the number of kilowatt isn't per inference session either, the number of Watt•hour is)

>It's just not part of the entended use-case, and hasn't been trained to do so. It's almost like complaining that StableDiffusion isn't good at text generation…

For the most part none of these systems were trained to do anything. The capabilities are emergent. My statement about the benchmarks remains unfazed.

>The biggest models will be abble to run on the CPU just fine with llama.cpp as long as you have enough (cheap) RAM. Sure it's slow, but you can run it.

Yes, but what products/clients are you pitching where that kind of wait will be acceptable? Time is money. Standing up racks of maxed out RAM server slots is still far from inexpensive too.

>What is that “per inference session” doing here? You pay the hardware only once you know… (and the number of kilowatt isn't per inference session either, the number of Watt•hour is)

Good job at completely missing the point. I know what a kilowatt is and it is the unit I meant, not kilowatt-hour. I'm referring to the hardware necessary to run one user session of inference. While user 1's tokens are generating, users 2 and up are in the queue waiting. If you want to serve multiple users simultaneously you will need to invest in multiple $xx,xxx units of hardware each requiring a multi-kW electrical circuit.

There is a reason ChatGPT incurs electrical bills on the order of a million dollars per day.

> For the most part none of these systems were trained to do anything. The capabilities are emergent.

The base models yes, but the chat version have been explicitly tuned for specific use-cases, it's not just basic next-token-prediction.

> Time is money

No, time is time, only context gives it monetary value: a single ms in some context can be much more expensive than an hour in another.

> Standing up racks of maxed out RAM server slots

Why are you talking about servers we're talking about the ability to run things on consumer hardware here…

> There is a reason ChatGPT incurs electrical bills on the order of a million dollars per day

Because they spend many MWh, to run all of their customers' requests, but when you're running your own requests on your own hardware, then the energy consumption is tiny, even if you needed a lot of instantaneous power.

>Llama really isn't open source, at least not in the sense of FOSS licenses like GPL or MIT.

Note that I never said "open source" just "open models". As in, I can now actually build things with GPT3 capability that run locally. I couldn't care less about the model code.

>Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session.

I'm running llama-2-7b-chat on my 8GB M1 Mac right now with llama.cpp. Completions are instant, and essentially at GPT3 levels of accuracy.

The higher param models require up to 64GB RAM, but it's all CPU based.

If you compiled llama.cpp with Metal I wouldn’t say its ~all CPU based. But it is still incredible! I need to set this up myself this weekend :)

https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML

> and $10,000+ of compute hardware per inference session.

What hardware would you need to run it at home?

>What hardware would you need to run it at home?

Step 1: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma...

Step 2: https://github.com/ggerganov/llama.cpp

Step 3: you're welcome

That's very helpful, thank you.

> and $10,000+ of compute hardware per inference session.

That is not true. A common macbook with lots of RAM (>32GB) is enough. Or any x86 computer with lots of RAM. llama.cpp is CPU only and quite fast

Reply on news.ycombinator.com