WizardLM 2 | Hacker News Reader

WizardLM 2

wizardlm.github.io

162 pointstosh63 comments

The 7B model is available on ollama if you want to try it: `ollama run wizardlm2` or `ollama run wizardlm2:7b`.

We're still crunching the 8x22B model to get it ready, and the 70B model isn't yet available.

If you can computationally afford it, 7b-q5_K_M is a way better choice. Default :7B goes to q4_0 which might give you subpar results.

Tried 7b q5 with some RAG tasks. Seems quite impressive.

And quantised gguf files by Bartowski if somebody wants to download and run through llama.cpp directly

https://huggingface.co/bartowski/WizardLM-2-7B-GGUF

Also gguf files by abroxis: https://huggingface.co/ABX-AI/WizardLM-2-7B-GGUF-IQ-Imatrix

For those adventurous souls who have gobs of memory, a good GPU, and plenty of disk space, some of the 8x22b models are now up. Use `ollama run wizardlm2:8x22b-q4_0`, `ollama run wizardlm2:8x22b-q8_0`, or `ollama run wizardlm2:8x22b-fp16`. The 4 bit quantized version is about 80GB, the 8 bit quantized version is about 150GB, and the non-quantized version is 281GB (!).

The 4bit quant doesn't seem to work for me, I keep getting: Error: exception create_tensor: tensor 'blk.0.ffn_gate.0.weight' not found

I've tried downloading it twice now.

Which version of Ollama are you using? You can check with `ollama -v`. Also, using `ollama list | grep wizardlm2` the 8x22b version should have ID `abda6e58fd1d`.

I ended up having to download the latest version directly from GitHub, and that fixed it. Looks like the 0.1.32 mac release hasn't been posted to your website yet.

I've been playing with the 8x22B version for several coding tasks today, and compared to DBRX and Command-R+, two other large and capable local models recently released, this WizardLM-2 is just a lot better. It follows the instructions better, and writes higher quality code, with little details that even ChatGPT 4 gets wrong.

Here's one example of the same prompt against the three models: https://twitter.com/blixt/status/1780191143747101072

(Running on an M3 Max 128GB it's quite fast – the times shown are including initial loading time which does take a while)

Deleted. Archive: http://web.archive.org/web/20240415221214/https://wizardlm.g...

https://github.com/nlpxucan/WizardLM/commit/4218c0ba64a84cfd...

Reason: https://twitter.com/WizardLM_AI/status/1780101465950105775 (they forgot toxicity testing before release and are doing it now)

a bit hidden but afaiu:

* 8x22B (based on Mistral 8x22B)

* 70B (based on llama2?)

* 7B (based on Mistral-7B-v0.1)

and the 70B model is not available yet on huggingface

https://huggingface.co/collections/microsoft/wizardlm-661d40...

Latest update: this link no longer works, leads to a 404 on GitHub. Folks on Reddit are reporting that it has disappeared from HuggingFace as well.

Does anyone know what is going on? Were there any announcements on its being unpublished?

Unpublished, they yanked everything off.

here's the 7B:

https://huggingface.co/amazingvince/Not-WizardLM-2-7B

apparently it's because they forgot to censor it: https://www.reddit.com/r/LocalLLaMA/comments/1c586rm/wizardl...

WizardLLM beats GPT4-0314 nearly close to the latest release GPT-4-1106-preview

By that benchmark, Claude Sonnet beats both.

And Claude Sonnet is great. It's what I recommend people use if they don't want to pay for an LLM or run one locally.

But Claude Sonnet is not open source and can't be run on your own hardware like Wizard.

How do you use Sonnet for free?

At https://poe.com/

Thanks!

Poe's Sonnet is limited to 15 free messages per day. The best freely accessible LLM with a generous daily allotment (300 msgs/day) is Bing Green Precise Mode. It's at about GPT4 level.

how are they offering this at discounts from the original? looks like the goal is to get subs to cover for the most frequent users?

Do you mean Poe or Microsoft? Microsoft is Microsoft. Poe has significantly reduced the generosity of their free tiers over time.

Poe, how are they undercutting what you can get from Microsoft/Anthropic directly...

How does the claude sonnet model on poe.com compare to the one on claude.ai?

On the official site Claude dot ao

On what benchmark? Did they publish benchmarks before deleting everything?

Mixtral8x22B looking very strong! Finetunes seem comparable to GPT4!

That's quite crazy to see that a model that's barely bigger than GPT-3 (and only uses a fraction of the compute due to its MoE architecture) can achieve such a thing.

It looks like the people who focasted that AI models would need to keep growing to improve their performance where completely misguided. I wonder if in 3 to 4 years we'll end up with models with less than 4B parameters we comparable performance as today's State of the Art.

It can also simply mean that benchmarks are not particularly representative of real-world performance on challenging reasoning tasks.

Of course they aren't, but it's still pretty evident that most opensource models are miles ahead of GTP-3, even the ones that are only a fraction of its size so there's still some massive improvement that doesn't depend on the model size itself.

GPT-3 sure. There's no model that's even close to GPT-4, though. And we don't really know much about the actual size of that, only lots of speculation.

My entire point since the beginning is that the models out there improved a ton compared to what used to be the state of the art just 4 years ago, with an equivalent when not much smaller number of parameters (GPT-3 was 175B params and on MMLU it performs worse than Mistral 7B [1]).

[1]: https://paperswithcode.com/sota/multi-task-language-understa...

They weren't misguided, they just over-focused on scaling parameters instead of appropriately scaling quality data in tandem.

A 4B has limited capacity to encode knowledge and algorithmic circuits. It's also too small to learn programs whose execution exceed the sizes of circuits it can encode. There is a hard cap on how much we can squeeze out of small models. What we need is better consumer hardware, so we don't have to hope for miracles.

Another hope is that the 1.58 bit/ternary quantization aware training of model scaling pans out. That'd be another axis of inefficiency beyond just parameter count.

> They weren't misguided, they just over-focused on scaling parameters instead of appropriately scaling quality data in tandem.

That's exactly what I mean by misguided, they were focusing on the wrong metric.

> A 4B has limited capacity to encode knowledge and algorithmic circuits. It's also too small to learn programs whose execution exceed the sizes of circuits it can encode. There is a hard cap on how much we can squeeze out of small models.

There must be some kind of cap indeed, but at this point we have no idea where it is, especially now that there are emerging infrastructures that aren't just transformers popping out.

> What we need is better consumer hardware, so we don't have to hope for miracles.

Hardware improvements are going to be a big factor, but there's no reason to think the software part (+ training data) isn't going to continue improving a lot.

Any idea what the context length is? Can't see it from the page linked.

I assume the same as the models that they're fine tuned off of.

For future reference, you can always check the `config.json` file in the repo for this info. Specifically `max_position_embeddings` is the key you're looking for.

7B: 32k 8x22B: 65k

Thanks!

What does "state-of-art" mean? Every one seems to be using this while in my head SOTA means "top of the line". When it says comparable results in benchmarks, how is it SOTA? Or does SOTA mean topping on really really specific benchmarks?

There's a table of MT-Bench scores. I presume we'll have to wait for the preprint before we know anything else.

How are folks running 8x22B (or MoE in general) in an affordable way?

For personal inference, llama.cpp can run some MoEs on CPU.

So like that.

seems to be roughly the same capacity as Command R+ if the arena leader board is to be trusted. I think Command R+ has fewer params?

I don't see WizardLM 2 on the leaderboard yet—are you looking at the previous version? Or do you have a link to a different leader board?

https://chat.lmsys.org

I said this based on the comparisons in the OP to other models which are on the leaderboard

WizardLM-2 8x22B appears to beat Command-R+ in their measurements.

I have just started experimenting with the 7B Q4 model by running "ollama run wizardlm2" and so far it seems to be very good at arithmetic word problems.

I usually use Mistral 7B-Q4 with Ollama's REST interface. I am curious about and looking forward to slightly adjust the prompts I use for wizardlm2 and see how it does.

While I want to be very optimistic about open models, the fact that almost all top open models heavily use GPT-4 for synthetic data is discouraging. Even now it is against TOS and over time I think over time OpenAI would become better at detecting meaning the gap between open and closed models would increase.

Considering these models are released by "Microsoft AI", I doubt they do anything against the ToS of "Open"AI.

I doubt it. They’re using gpt-4 data because it’s cheaper than getting real data, not because it’s better.

If openai cares enough to stop people scraping responses, then we’ll just crowdsource them like open assistant did with their dataset.

There is some news OpenAI spent 4 billion dollars for data sourcing.

Yeah but that’s what OpenAI might’ve spent

What can OpenAI realistically do about it except block the user account?

I think the TOS should have no effect on the license and copyright of the resulting model, since outputs of AI Models cannot be copyrighted (at least in the US).

OpenAI would likely start using AI to detect the content generation and ban the users. It wouldn't even need to be GPT level. Just they could use some AI labelling and counting.

Where do people check the leaderboard of the best performing LLM? The latest news I had was the best perfoming was claude.ai.

A good leaderboard is https://arena.lmsys.org/ (the "Leaderboard" tab).

Can I run this locally with a 4090?

If so, what’s the easiest way?

Even a 4-bit quant version of the MoE 8x22 is going eat ~80GB of VRAM. No way you're running this on a 4090 without setting it up as a GGUF to split between VRAM and regular RAM, and then you're going to have to deal with the low token rate as a result.

Running WizardLM-2 70B or lower WizardLM-2 7B is much more feasible however.

Look into Ollama:

https://ollama.com/library/wizardlm2/tags

They are based on llama2 architecture which performs a lot worse.

Reply on news.ycombinator.com