Hacker News Reader

Comment by smcleod | Hacker News Reader

Mistral and its hybrids are a lot better than GPT3.5, and while not as good as GPT4 in general tasks - they’re extremely fast and powerful with specific tasks. In the time it takes GPT4 to apologise that it’s not allowed to do something I can be three iterations deep getting highly targeted responses from mistral - and best yet - I can run it 100% offline, locally and on my laptop.

View on HN

There is an attempt to quantify subjective evaluation of models here[1] - the "Arena Elo rating". According to popular vote, Mistral chat is nowhere near GPT 3.5

[1] https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Doesn't seem like that's the mixture of experts model in the list? Or am I blind

Indeed, that is a previous Mistral model.

Starling-7B, OpenChat, OpenHermes in that table are Mistral-7B finetunes, and are all above the current GPT-3.5-Turbo (1106). Note how these tiny 7B models are surrounded by much larger ones.

Mixtral 8x7B is not in there yet.

ELO takes a while to establish. It does not sound likely that the newer GPT3.5 is that much worse than the old one that has a clear gap to all the non proprietary models. In the immediate test, GPT-3.5 clearly outshines these models.

> ELO takes a while to establish.

Well, Starling-7B was published two weeks ago; GPT-3.5-turbo-0613 is more than a month old snapshot, which should probably be enough time. OpenChat and OpenHermes are about a month old as well.

>It does not sound likely that the newer GPT3.5 is that much worse than the old one

In fact, this version received complaints almost immediately. https://community.openai.com/t/496732

>In the immediate test, GPT-3.5 clearly outshines these models.

It might be so, but it's not clear to me at all. I tested Starling for a bit and was really surprised that it's a 7B model, not a 70B+ one or GPT-3.5.

I disagree - lmsys score for new chatgpt has been relatively constant, and OAI is probably trying to distill the model even further.

Not my intent to argue about data at any point in time but note that as of today gpt-3.5-turbo-0613 (June 13th 2023) scores 1112, above OpenChat (1075) and OpenHermes(1077).

That's not much relative difference. How much does 1% difference make?

I am tempted to call it equivalent.

nah that is sizeable

I guess one thing people have learned is that these small differences whatever benchmark turn out to be huge differences qualitatively.

Are they a lot better than 3.5? I see wildly varying opinions.

Mistral-Medium, the one announced here which beats GPT-3.5 on every benchmark, isn't even available yet. Those opinions are referencing Mistral-Tiny (aka Mistral-7B).

However, Mistral-Tiny beats the latest GPT-3.5 in human ratings on the chatbot-arena-leaderboard, in the form of OpenHermes-2.5-Mistral-7B.

Mixtral 8x7B aka (Mistral-Small) was released a couple of days ago and will likely come close to GPT-4, and well above GPT-3.5, on the leaderboards once it has gone through some finetuning.

They could be.

It is an open question whether the driving force will be OSS improving or OAI continuing to try to distill their model.

I was going to ask about this. So these open models are uncensored and unconstrained?

What laptop are you using to run which model, and what are you using for that?

M2 MacBook Pro, I run man different models but mistral, zephyr, deepseek. I use Ollama and LM Studio.

The easiest way is to use ollama - mistral 7b, zephyr 7b, openhermes 2 are all decent models, I guess in fact openhermes 2 can do function calling.

If you further want a smaller one, stablelm-zephyr 3b is a good attempt with ollama.

Bit of a stretch to say they're a lot better when you look at evals

They are not better than GPT 3.5 except for some of the public benchmarks. Also they are not faster than GPT 3.5. And they are not cheaper if you run finetuned model for specific task.

Reply on news.ycombinator.com