LLM Falcon 180B Needs 720GB RAM to Run | Hacker News Reader

LLM Falcon 180B Needs 720GB RAM to Run

kaitchup.substack.com

102 pointsbehnamoh35 comments

Interesting article, but the premise is wrong. You don't need 720GB ram to load the model. The author claims that you need to first load up the model (360GB) and then the load the weights (another 360 GB) and the replace the first with the latter. HF allows initializing empty models that do not take up any space using accelerate, especially so that you don't run into that problem.

Still, 180B is a best to load up.

later he said HF's tensor has almost zero copy, so 720GB is not needed using this format

No, I think the author gets this mixed up.

Zero copy does not play a role here - the problem is how the model is being loaded in the steps that are shown in the article and they have nothing to do with Safe tensors' zero copy ability. Check out the article by Hugging face to see how to do it with only 360GB instead of 720:

https://huggingface.co/docs/accelerate/usage_guides/big_mode...

> Still, 180B is a best to load up.

Best -> Beast

Title here seems clickbait.

The title from the article I see it now is "Falcon 180B: Can It Run on Your Computer?"

The content does mention 720GB RAM, but quickly clarifies that it can be reduced:

QUOTE:

Step 1 and step 2 are the ones that consume memory. In total, you would need 720 GB of memory available. This can be CPU RAM, but for fast inference, you may want to use GPUs, e.g., 9 A100 with 80 GB of VRAM. This wouldn’t be enough for batch inference.

With either CPU RAM or VRAM, that’s a lot of memory. Fortunately, these requirements can be easily reduced.

Current LLMs are in the Pentium IV Netburst stage: hypertrophy of components on old archs have gotten to the limits of what is physically feasible. Get ready for the next phase of leaner models.

It makes no sense to me why people want to run this model when it barely outperforms Llama 2 while being over twice the size.

I was going to say: because Falcon 180GB is open source, so you can run it for any purpose, without worrying about license restrictions.

However, it transpires Falcon 180B uses a proprietary license “based upon” the Apache license. Despite Falcon's claims to the contrary, it is not open source.

And that's disappointing because Falcon 40B is actually open source (and Apache licensed).

Yep. No truly open source big models around rn, but people keep calling them open source. Some amazing marketing by meta.

I am not sure how can a binary have open source license at first place

not so much about reading/modifying the source, more about what you can build on top, but point taken, could use a different word, "public domain" would be nice if anyone actually wanted to give something back to the public after digesting all the information ever produced

Source code is, for the GPL, the “preferred form for modification”. Given the outrageous cost of training from scratch, plus the ability to fine tune models... I don't think it's completely ridiculous.

Edit: plus, the industry acts like models aren't derivative works of the training data. If true the source license is less important than the model!

Through the magic of marketing

At least Meta's license mainly restricts usage from other large companies.

The Falcon 180B license is special, even in the context of "open" LLM models:

https://huggingface.co/spaces/tiiuae/falcon-180b-license/blo...

> 5.3. The Acceptable Use Policy may be updated from time to time. You should monitor the web address at which the Acceptable Use Policy is hosted to ensure that your use of the Work or any Derivative Work complies with the updated Acceptable Use Policy.

This is basically "don't use it" for any business that retains a lawyer.

Meta’s llm license prevents you from using llama data to train or fine tune a non llama model.

Feels very EEE to me.

> because Falcon 180GB is open source, so you can run it for any purpose, without worrying about license restrictions.

This isn't as big of a deal as people try to claim it is.

LLama 2 can be used for almost anything unless you are like Google or Apple.

But if you don't work there, then you are basically safe to do almost anything with it.

You can't use it for generating training data.

Quantized to 8 bits it needs only 178GB to run. With llama.cpp on a dual 7763 inference runs at 0.72 tokens per second (maybe about 32 words per minute), without any tuning effort.

[A chat between a curious user and an artificial intelligence assistant named Falcon. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Please give a short explination of 'clickbait'.\nFalcon:] Certainly! Clickbait is a term used to describe online content that is designed to attract attention and encourage users to click on a link to a particular webpage. It often uses sensational or provocative headlines, images, or videos to entice users to click through to the content. The goal of clickbait is typically to generate more traffic to a website and increase advertising revenue. However, it can sometimes be misleading or deceptive, as the actual content may not live up to the expectations set by the clickbait headline or image.

Seriously, my self reply showing output from Falcon 180B is flagged??

I think 100% LLM-generated comments are fairly unpopular.

Suggestion: if you feel the need to share some LLM output with HN, prefix it with a human-written sentence or two explaining why it's worth reading.

Llama-2-70b with 16-bit precision = 2 bytes * 70 billion = 140 GB of memory

Assuming zero copy, it should be loadable on around 170GB.

Reference: https://www.hopsworks.ai/dictionary/llms-large-language-mode...

No one runs inference at full FP16 precision. 6 bit quant gives ~90% of original quality at more than half memory savings.

Sigh. My comment gives a measurement of the memory usage of the model in question, Falcon 180B, (in mostly-q8 form, which is how someone would likely use it if not q4).

Perhaps it would have been more obvious that I actually ran the model and measured it if people hadn't flagged the self-reply with the model output.

A 4-bit quantized version will require (still a whopping) ~100GB of RAM using tools like Ollama and Llama.cpp

If you want to try it with Ollama on macOS (keeping in mind you'll need the new Mac Studio with 192GB of memory) this command will work:

  ollama run falcon:180b

Right now it gets about ~5 t/s.. there's quite a bit of work being done to improve inference speeds like speculative sampling which uses a smaller model for a subset of tokens

Surely the next step will be sparsity and other optimizations beyond brute aggressive quantization, maybe an MoE model like OAI is doing

> brute aggressive quantization

Quantization like ExLlama's EX2 or even llama.cpp's k-quants is already far from brute force: https://github.com/turboderp/exllamav2#exl2-quantization

> The format allows for mixing quantization levels within a model to achieve any average bitrate between 2 and 8 bits per weight. Moreover, it's possible to apply multiple quantization levels to each linear layer, producing something akin to sparse quantization wherein more important weights (columns) are quantized with more bits. The same remapping trick that lets ExLlama work efficiently with act-order models allows this mixing of formats to happen with little to no impact on performance. Parameter selection is done automatically by quantizing each matrix multiple times, measuring the quantization error (with respect to the chosen calibration data) for each of a number of possible settings, per layer. Finally, a combination is chosen that minimizes the maximum quantization error over the entire model while meeting a target average bitrate.

Llama.cpp is also working on a feature that let's a small model "guess" the output of a big model which then "checks" it for correctness. This is more of a performance feature, but you could also arrange it to accelerate a big model on a small GPU.

> You will need at least 400GB of memory to swiftly run inference with Falcon-180B.

[0] https://huggingface.co/tiiuae/falcon-180B

That is 180 billion parameters not 180 bytes, for anyone else confused.

Yep and 16 bits per parameter means 360 GiB, then because you typically first allocate an empty model then separately load the weights, 720 GiB.

On a side note, has anyone here used this model for anything's interesting? Can it outperform popular smaller models?

Given it's ridiculous license, how can we? I'd rather accept meta's proprietary "open source" license that deal with that nonsense.

Eventually, other (real) open source models will reach this size as well. I was just wondering about the technical aspects of a model this big. Does performance escalate? Does it saturate quickly around 70B? Something like that.

I don't really understand all the fuss about license agreements in this space. Just ignore their stupid license and obtain the model weights via alternative means. Both LLaMA and LLaMA v2 are readily available on BitTorrent[0].

It's not clear to me whether compilations of model weights are really copyrightable in most jurisdictions. Even if they were, enforceability is highly questionable. Anyway downloading is not the same as uploading; the recipient is not the one making the copy.

So long as you are not redistributing the model weights but just using those model weights to do something, I don't see how you're violating copyright. If you pirate a compiler and distribute your own software compiled with it--is that compiled software a derivative work of the compiler? Is my proprietary software a derivative work of g++ because I used it to compile my software?

Even providing a direct API to inference with or finetuning a model should be OK. I can't see how the output of inference could be a derivative work of the model, which is only part of the input along with the hyperparameters, seed value, potential fine-tuning data, and the prompt. If, alternatively, it was, then the output of training (i.e. the model itself) is surely a derivative work of the input--including my data they slurped up!

On the other hand, redistributing finetuned models might need to be done by weight diffs, as those probably are derived works.

[0]: e.g. b8287ebfa04f879b048d4d4404108cf3e8014352 and 14ec42c98d0b12f5b12f5b3d670cdf4915c1feaf

Your proprietary software won't be a derivative work of most of G++, but it will likely be a derivative work of libgcc (compiler runtime), libc (on Linux), and libstdc++.

Anyway, running copyrighted software without a valid license is clear cut unlawful, at least in the EU. That's because running the application requires copying it (into RAM).

No, really, the law is that pedantic. "Loading" is a restricted act under Article 4 of the Computer Programs Directive.

Reply on news.ycombinator.com