OpenAI is going to be left in the dust by (actual) open models. Llama 2 is already reaching GPT-3 levels, and can run inference on consumer hardware. Crazy how fast that flipped.
There's a huge gap between GPT-3.5 and 4, put there by a massive amount of money, from my understanding.
To compete, with open source projects being less well funded, I would assume that orders of magnitude improvements in training cost would be required. What do you see driving this, and who do you see paying for it?
If Meta, or anyone else, gets something that beats GPT-4, I would naively assume they would monetize it. If someopen source effort manages to beat GPT-4, then I assume some well funded, profit seeking, entity will dump orders of magnitude more money into whatever enabled the open source offerings to compete.
I don't think GPT-4 is the pinnacle of OpenAI, or that their funding will run dry, so they shouldn't be viewed as a static entity.
My bet is that Meta has pivoted almost entirely to this space with their R&D in the last six months. Llama 2 is spectacular. And with its' success, there will undoubtedly be more. They also happen to have access to limitless amounts of compute, cash, and engineering that puts OpenAI to shame. This could finally be their chance to create a platform for real. And the open source community and startups will benefit off of that.
Owning a platform. Zuck's dream.
They put tons of money into these models, nurture an ecosystem of companies built around them, and then start gradually figuring out a licensing model for the ones that take off.
Furthermore, Llama remains well below GPT-3 on human rated tests such as programming, and GPT-3 is already over three years old. It is also misleading to suggest Llama 2 can be ran on consumer hardware - the smaller and quantized models can but those are even more lacking in capability. Full power Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session.
OpenAI does not have a moat, but they do have a very high wall.
What hardware would you need to run it at home?
Step 1: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma...
Step 2: https://github.com/ggerganov/llama.cpp
Step 3: you're welcome
That is not true. A common macbook with lots of RAM (>32GB) is enough. Or any x86 computer with lots of RAM. llama.cpp is CPU only and quite fast
Note that I never said "open source" just "open models". As in, I can now actually build things with GPT3 capability that run locally. I couldn't care less about the model code.
>Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session.
I'm running llama-2-7b-chat on my 8GB M1 Mac right now with llama.cpp. Completions are instant, and essentially at GPT3 levels of accuracy.
The higher param models require up to 64GB RAM, but it's all CPU based.
This is not what the Llama 2 license says [0]. There is a cap on the number of active users of products (any products, not just ones that may make use of Llama) by the company who plans to use Llama 2 as of Llama 2 release date.
Not a “future cap on number of users of your Llama2-based product”. Also, the cap is 700 million users.
> If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee's affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta
[0] https://github.com/facebookresearch/llama/blob/main/LICENSE
It's just not part of the entended use-case, and hasn't been trained to do so. It's almost like complaining that StableDiffusion isn't good at text generation…
> It is also misleading to suggest Llama 2 can be ran on consumer hardware - the smaller and quantized models can but those are even more lacking in capability
The biggest models will be abble to run on the CPU just fine with llama.cpp as long as you have enough (cheap) RAM. Sure it's slow, but you can run it.
> Full power Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session.
What is that “per inference session” doing here? You pay the hardware only once you know… (and the number of kilowatt isn't per inference session either, the number of Watt•hour is)
For the most part none of these systems were trained to do anything. The capabilities are emergent. My statement about the benchmarks remains unfazed.
>The biggest models will be abble to run on the CPU just fine with llama.cpp as long as you have enough (cheap) RAM. Sure it's slow, but you can run it.
Yes, but what products/clients are you pitching where that kind of wait will be acceptable? Time is money. Standing up racks of maxed out RAM server slots is still far from inexpensive too.
>What is that “per inference session” doing here? You pay the hardware only once you know… (and the number of kilowatt isn't per inference session either, the number of Watt•hour is)
Good job at completely missing the point. I know what a kilowatt is and it is the unit I meant, not kilowatt-hour. I'm referring to the hardware necessary to run one user session of inference. While user 1's tokens are generating, users 2 and up are in the queue waiting. If you want to serve multiple users simultaneously you will need to invest in multiple $xx,xxx units of hardware each requiring a multi-kW electrical circuit.
There is a reason ChatGPT incurs electrical bills on the order of a million dollars per day.
The base models yes, but the chat version have been explicitly tuned for specific use-cases, it's not just basic next-token-prediction.
> Time is money
No, time is time, only context gives it monetary value: a single ms in some context can be much more expensive than an hour in another.
> Standing up racks of maxed out RAM server slots
Why are you talking about servers we're talking about the ability to run things on consumer hardware here…
> There is a reason ChatGPT incurs electrical bills on the order of a million dollars per day
Because they spend many MWh, to run all of their customers' requests, but when you're running your own requests on your own hardware, then the energy consumption is tiny, even if you needed a lot of instantaneous power.