Hacker News Reader

Hacker News

Comment by byteknight | Hacker News Reader

Full thread

byteknight

First test I tried to run a random taxation question through it

Output: https://gist.github.com/IAmStoxe/7fb224225ff13b1902b6d172467...

Within the first paragraph, it outputs:

> GET AN ESSAY WRITTEN FOR YOU FROM AS LOW AS $13/PAGE

Thought that was hilarious.

View on HN

The `mixtral:8x22b` tag still points to the text completion model – instruct is on the way, sorry!

Update: mixtral:8x22b now points to the instruct model:

  ollama pull mixtral:8x22b
  ollama run mixtral:8x22b

Wait. Isn't it a breaking change to change the underlying model like this? Wouldn't people start running into consistency issues in production? (given ollama appears to be oriented towards backend use)

Sure, in theory. But if you move so fast that you already are running the base 8x22B model from last week, you can easily fix this.

I've long thought that if you want reproducibility and reliability, you need to pin your deps.

So, IMO, the change is very much worth it to reduce confusion going forward.

That's not the model this post is about. You used the base model, not trained for tasks. (The instruct model is probably not on ollama yet.)

I absolutely did not:

ollama run mixtral:8x22b

EDIT: I like how you ninja-editted your comment ;)

Considering "mixtral:8x22b" on ollama was last updated yesterday, and Mixtral-8x22B-Instruct-v0.1 (the topic of this post) was released about 2 hours ago, they are not the same model.

Are we looking at the same page?

https://imgur.com/a/y6XfpBl

And even the direct tag page: https://ollama.com/library/mixtral:8x22b shows 40-something minutes ago: https://imgur.com/a/WNhv70B

Let me clarify.

Mixtral-8x22B-v0.1 was released a couple days ago. The "mixtral:8x22b" tag on ollama currently refers to it, so it's what you got when you did "ollama run mixtral:8x22b". It's a base model only capable of text completion, not any other tasks, which is why you got a terrible result when you gave it instructions.

Mixtral-8x22B-Instruct-v0.1 is an instruction-following model based on Mixtral-8x22B-v0.1. It was released two hours ago and it's what this post is about.

(The last updated 44 minutes ago refers to the entire "mixtral" collection.)

I get:

ollama run mixtral:8x22b

Error: exception create_tensor: tensor 'blk.0.ffn_gate.0.weight' not found

You need to update ollama to 0.1.32.

Thanks. That did it.

And where does it say that's the instruct model?

Yeah this is exactly what happens when you ask a base model a question. It'll just attempt to continue what you already wrote based off its training set, so if you say have it continue a story you've written it may wrap up the story and then ask you to subscribe for part 2, followed by a bunch of social media comments with reviews.

It can be fun, though, to prompt a text completion with something like "I'm thinking about" and just seeing what random thing it completes it with.

Not instruct tuned. You're (actually) "holding it wrong".

Looks like an issue with the quantization that ollama (i.e llama.cpp) uses and not the model itself. It's common knowledge from Mixtral 8x7B that quantizing the MoE gates is pernicious to model perplexity. And yet they continue to do it. :)

No, it's unrelated to quantization, they just weren't using the instruct model.

Reply on news.ycombinator.com