Not sure who this is aimed at? The avg programmer probably doesn’t have the gear on hand to run this at the required pace
Cool nonetheless
Cool nonetheless
Anybody have $10 billion sitting around to deploy that gigantic open source set-up for millions of users? There's your moat and only a relatively few companies will be able to do it.
One of Google's moats is, has been, and will always be the scale required to just get into the search game and the tens of billions of dollars you need to compete in search effectively (and that's before you get to competing with their brand). Microsoft has spent over a hundred billion dollars trying to compete with Google, and there's little evidence anybody else has done better anywhere (Western Europe hasn't done anything in search, there's Baidu out of China, and Yandex out of Russia).
VRAM isn't moving nearly as fast as the models are progressing in size. And it's never going to. The cost will get ever greater to operate these at scale.
Unless someone sees a huge paradigm change for cheaper, consumer accessible GPUs in the near future (Intel? AMD? China?). As it is, Nvidia owns the market and they're part of the moat cost problem.
You don't have to run them locally.
It is in at least 2025. AMD (and Intel, maybe) will have M-Pro-Esque APUs that can run a 70B model at very reasonable speeds.
I am pretty sure Intel is going to rock the VRAM boat on desktops as well. They literally have no market to lose, unlike AMD which infuriatingly still artificially segments their high VRAM cards.
Models of any given quality are declining in size (both number of parameters, and also VRAM required for inference per parameter because quantization methods are improving.)
Raw FLOPs may increase each generation but VRAM becomes a limiting factor. And fast VRAM is expensive.
I do expect to see incremental innovation in reducing the size of foundational models.
for end users, yes. For small companies that want to finetune, evaluate and create derivatives, it reduces the cost by millions.
I run 33B parameter models on my RTX 3090 (24GB VRAM) no problem. 70B should easily fit into 64GB of RAM.
In traditional software, the same program compiled for 32-bit and 64-bit architectures won’t be able to handle all of the same inputs, because the 32-bit version is limited by the available address space. It’s still the same program.
If we’re not willing to declare that you are a completely separate person when you’re tired, or that 32-bit and 64-bit versions are completely different programs, then I don’t think it’s worth getting overly philosophical about quantization. A quantized model is still the same model.
The quality loss from using 4+ bit quantization is minimal, in my experience.
Yes, it has a small impact on accuracy, but with massive efficiency gains. I don’t really think anyone should be running the full models outside of research in the first place. If anything, the quantized models should be considered the “real” models, and the full fp16/fp32 model should just be considered a research artifact distinct from the model. But this philosophical rabbit hole doesn’t seem to lead anywhere interesting to me.
Various papers have shown that 4-bit quantization is a great balance. One example: https://arxiv.org/pdf/2212.09720.pdf
The question of whether I am still me after a traumatic brain injury is philosophically unclear, and likely depends on specifics about the extent of the deficits.
It’s far more similar to the model being perpetually tired than it is to a TBI.
You may nitpick the analogy, but analogies are never exact. You also ignore the other piece that I pointed out, which is how we treat other software that comes in multiple slightly different forms.
The difference can be measured empirically, but is it noticeable in real world usage.
As someone else pointed out, FLAC is objectively more accurate than mp3, but how many people can really tell? Is it worth 3x the data to store/stream music in FLAC?
The quantized model would run at probably 4x the speed of the unquantized model, assuming you had enough memory to choose between them. Is speed worth nothing? If I have to wait all day for the LLM to respond, I can probably do the work faster myself without its help. Is being able to fit the model onto the hardware you have worth nothing?
In essence, quantization here is a 95% “accurate” implementation of a 67% accurate model, which yields a 300% increase in speed while using just 25% of the RAM. All numbers are approximate, even the HumanEval benchmark should be taken with a large grain of salt.
If you have a very opulent computational experience, you can enjoy the luxury of the full 67.8% accurate model, but that just feels both wasteful and like a bad user experience.
Mixtral runs at about 43 tokens/s at q3_K_S with all layers offloaded. I normally avoid going below 4-bit quantization, but Mixtral doesn’t seem phased. I’m not sure if the MoE just makes it more resilient to quantization, or what the deal is. If I run it at q4_0, then it runs at about 24 tokens/s, with 26 out of 33 layers offloaded, which is still perfectly usable, but I don’t usually see the need with Mixtral.
Ollama dynamically adjusts the layers offloaded based on the model and context size, so if I need to run with a larger context window, that reduces the number of layers that will fit on the GPU and that impacts performance, but things generally work well.
There are a few things to keep in mind: no programmer that I know is sitting there typing code for hours at a time without stopping. There’s a lot more to being a developer than just typing, whether it is debugging, thinking, JIRA, Slack, or whatever else. These CoPilot-like tools will only activate after you type something, then stop for a defined timeout period. While you’re typing, they do nothing. After they generate, they do nothing.
I would honestly be surprised if the GPU active time was more than 10% averaged over an hour. When actively working on a large LLM, the RTX 3090 is drawing close to 400W in my desktop. At a 10% duty cycle (active time), that would be 40W on average, which would be 320Wh over the course of a full 8-hour day of crazy productivity. My electric rate is about 15¢/kWh, so that would be about 5¢ per day. It is absolutely not running at a 100% duty cycle, and it’s absurd to even do the math for that, but we can multiply by 10 and say that if you’re somehow a mythical “10x developer” then it would be 50¢/day in electricity here. I think 5¢/day to 10¢/day is closer to reality. Either way, the cost is marginal at the scale of a software developer’s salary.
Ollama caches the last used model in memory for a few minutes, then unloads it if it hasn’t been used in that time to free up VRAM. I think they’re working on making this period configurable.
Latency is very good in my experience, but I haven’t used the local code completion stuff much, just a few quick experiments on personal projects, so my experience with that aspect is limited. If I ever have a job that encourages me to use my own LLM server, I would certainly consider using it more for that.
Note that M1/M2 Ultra is quite a bit faster than M3 Max, mostly due to 800 Gb/s vs 400 Gb/s memory
For example, take a look at the GGUF file sizes here: https://huggingface.co/TheBloke/Llama-2-70B-GGUF
Could you please expand on what else would be capable of running such models locally?
How about a linux laptop/desktop with specific hardware configuration?
Compute is actually not that big of a deal once generation is ongoing, compared to memory bandwidth. But the initial prompt processing can easily be an order of magnitude slower on CPU, so for large prompts (which would be the case for code completion), acceleration is necessary.
For example both the RTX 4090 and the RTX 6000 Ada Generation use the AD102 chip. The RTX 6000 Ada though, would be able to run 70b models due to the larger memory pool despite having the same memory interface width.
It also really depends on what you consider "beginner level". Fine-tuning is really easy these days and many people do it, but you really want CUDA for that.
People are recommending Macbooks because they're a relatively cheap and easy way to get a very large amount of RAM hooked up to your accelerator.
Note that these are quantized versions of the model, so they're not as good as the original 70B model, though people claim their performance is really close to original performance. To run without quantization you'd need about 140GB of VRAM. Which would only be possible with an NVidia H100 (don't know the price) or two A100's (at $18,000 each).
Instructions here: https://github.com/facebookresearch/llama/pull/947/
Or how you deal with context length? I.e. do you send anything other than the current file? How is the prompt constructed?