Falcon 40B LLM (which beats Llama) now Apache 2.0

This will require a cpp port first to run on e.g. apple silicon?

The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?

is there a model version where it can be deployed using huggingface's tool?

How is it possible to run this model on 2x 4090s ?

I thought that 4090s were "nerfed" and nvlink support removed - https://www.windowscentral.com/hardware/computers-desktops/n...

How can it be deployed on a Huggingface Space or Colab notebook?

Very cool but how does this compare to GPT-4 (before it was nerfed)?

I feel like the best benchmark atm is the orig gpt-4 version.

How does it compare to OpenAI's GPT4? It's not listed in the table.

Probably PR to attract subsidies/investment. UAE might be throwing more money on tech/innovation branded projects than China.

maybe attention IS all you need

Not sure I care to be honest. With an open license the community can now take this and roll with it.

What PR noise they make is secondary to that in my mind.

Do you consider the Unity model "sleazy"? But I agree that it was poor form to call it open-source at the time. Sadly, it seems to be standard practice in the LLM space to release models with all kinds of restrictions on use and call them "open source".

Well now Falcon is open source. Given they are giving away something that was very expensive to train, I am grateful.

Well if its Apache 2.0 now, can't you clone it as-is and have it be open "forever?"

This is such a bizarre comment to make about something that is Apache licensed. Who cares what happened before now that it’s truly open?

Yann LeCun: "No. But it's not because we don't want to. It's because of complicated legal issues."

https://twitter.com/ylecun/status/1651782621540524032

Meta's eventual, internal LLM architecture will be totally different than the open source LLMs, right? They don’t need to run on commodity hardware.

Maybe I am cynical, but I dont see the incentive for Meta to contribute an open model.

Research in the LLM space is moving so fast, I doubt any architecture will "stick".

Why do people think that Meta released their model in order to get open source coders to improve their models? They will get absolutely no competitive advantage from this. Every other team developing a closed source LLM can easily copy the innovations that open source coders have applied to Llama on their own, closed source models.

There's no advantage here. Meta just spent $10 million on releasing fun chaos into the world and increasing their recruiting power.

Doesn't one need to have a bunch of "very good" data to train on? I'm under the impression that sourcing costs are large.

I'm running Vicuna on a free 4core Oracle VPS, and it's perfectly usable for a Discord bot. Responses rarely take more than 15 seconds with <256 max token limit, and the responses are much more entertaining than GPT 3.5. I'm not using the streaming API my server software[0] offers, but if I did it would probably load somewhere between the speeds of GPT-3.5 and GPT-4. It's more or less the same time a human would take to compose the same message.

So... not exactly a serious use-case. But it's what I'm using, and now I'm saving 10s of dollars on inferencing costs per month!

[0] https://github.com/go-skynet/LocalAI

I'm also using this to improve acceleration - https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...

Fine-tuning is a much better proposition than you’re giving it credit for. Papers are coming out demonstrating that 7B parameter models can outperform GPT-4’s quality when trained on a limited set of tasks. Yet, a 7B model offers comparatively cheap and fast inference. Furthermore, for a lot of use cases, few-shot prompting is infeasible because you need to supply 2-3k tokens worth of few-shot examples with every prompt in order to fully specify the behavior you want. (As an example, think of long-form summarization where you want the summary to adhere to certain rules.)

>From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries.

That's because we're only half a year into LLMs becoming mainstream. Give it 3-4 years. The advancements in bringing down model size, optimizations, and newer GPUs, SoCs from Nvidia, AMD, Apple, Intel, Qualcomm, etc will make it so that top LLMs will run on a highend laptop/desktop.

This is bleeding edge stuff.

All advances in this direction do indicate that it will be easier and easier for more people to do things with it.

This doesn't need to work for everyone.

A 4090 costs today 2k, the 3090 with also 24gb costs today 1k and costed 2k.

> HF requires you to use their commercial offerings

Some of which are quite affordable ($80 per month). Larger ones can be like 2000 a month which is still ok to prototyping phase. You're basically paying for aws/gcp infrastructure.

I quite liked the UX of it, very intuitive. My trouble was finding a model that executes out-of-the-box tho. All of the GPT ones crash on startup.

70GB of RAM would cost around $150 these days depending on how you get there. 64GB (2x32GB) of DDR4 is around $140 then another 8GB stick would be around $15.

Used DDR3 ECC would be roughly half that.

With 4-bit quantization should take less than 40GB of VRAM/RAM.

That much good RAM in itself isn't super expensive. So does the rest of the hardware have to be particularly powerful?

I briefly looked at prices a few days ago, I think you can rent an 80gb GPU for about $2.50 an hour. Then you just pay while you're using it.

Someone else might be better able to confirm the pricing, but in any case you don't need to purchase the hardware.

Some people use second hand P40 GPUs, which go for around 200-300$. Combine 3 of them with SLI and you've got 72GB of VRAM for less then $1000

> https://github.com/ggerganov/llama.cpp/issues/1602#issuecomm...

Here somebody quantized it down to 29929.56MB .

Stupid question but for feed-forward models why do we not yet have some kind of CPU RAM memory swap mechanism? Why is Pytorch still trying to load the whole damn model into GPU RAM at once and then complaining when it can't, instead of swapping portions of the model to CPU RAM, or hell, even SSD?

Sure, it might be a lot slower, but that's a lot better than "I give up, go buy $20K worth of hardware"

Same question, so far I've found this thread https://github.com/ggerganov/llama.cpp/issues/1602 where people work on it.

It would as open source improvements would start to exceed performance of 3.5 for specific use-cases. At the very least they would have to make it fine-tunable.

Did Stable Diffusion "force" OpenAI or Google to open-source their imaging models? No. Don't see what the mechanism would be for that

No.

There no reason, particularly, to believe either that it does, or it would.

For openai to scramble and try to “catch up” with a competitor and make such a massive change in strategy would require someone to be offering an equivalent service (hosted inference) that was either orders of magnitude cheaper than their offering and just as good, or significantly better than it. Or legal compulsion.

This is none of those things. They won’t care.

At the most basic, you give it text and it can guess at what comes next. This means you could type "10 types of ferns" and it will build a list of 10 ferns. Or you could type it out how a transcript of a conversation would look like and it will basically fill in the other "side" of the conversation to make a chatbot (all the complicated chatbots are basically abstracting this). Think of it like a text box with a super-smart (arguably) person also having access where you can type one thing and then it'll type whatever it thinks would be the next thing to type.

As with everything, follow the money, pretty much.

There will be an ASIC as soon as serious money is being made from LLMs, most use cases atm seem to be in prototype/toy stage, but I imagine we'll start seeing that change.

Can you give examples?