Hacker News Reader

Comment by dontupvoteme | Hacker News Reader

Mistral IMO:

1. Doesn't have to pay the obscene alignment tax that OpenAI/etc have to (something between 20-50% of from what i understand) nor worry nearly as much about """the brand""" and the baggage which comes with that.

2. Likely will have/has the EU behind them out of pragmatic protectionism realpolitik verus the US (which basically started with GDPR) - though it might end up being a duo with Aleph Alpha.

3. Generally has the widespread support of the hobbyist/opensource community (for whatever aims those may be), both out of ethical/moral consideration and performance/quality reasons

4. Seems already *quite* competitive with GPT3. I rarely find i need to invoke GPT4 and when I have to, I'm annoyed with the latency and milquetoastian nature of the damn thing.

Wish I could purchase stock in these guys honestly

View on HN

Mistral fine-tunes are competitive with GPT 3.5 and 4 on many, many tasks, people are just enamored with the (admittedly impressive) ability of very large general models. In terms of business applications, the range of things that need GPT-4 to solve vs. Mistral 7B is diminishing small and infinitesimal when you factor in Mixtral. I’m all for doing advanced research to create the most interesting and powerful thing possible, but in more practical terms, OpenAI has zero moat. Once the infrastructure around open source stabilizes and we get a few more rounds of improvement to efficiency, I think this will be plainly obvious.

Do you run the model locally on your computer or how do you use it as an alternative to ChatGPT?

I can run Mistral 7B, in 3-bit quantization, on my phone, and I get helpful answers for (just today) seeing environment variables in Python.

What do you use to run it on your phone?

The iOS app “MLCChat”

I have a linux box which i run it locally for batch processing, otherwise i just run it through poe (serves as a good comparative website endpoint for differently models i find.)

Can you elaborate what poe is?

https://huggingface.co/chat

chat.lmsys.org has a zillion models you can use.

Check out LMStudio if you want to try. Open source and quite easy to use, doesn't need a beefy machine.

Thank you, it looks really good

"milquetoastian nature"?

I had to lookup for that too: milque·toast: A timid or feeble person.

white bread

prudish

square

no fun

sorry, i apologize for using the term milquetoast without context ;)

it just kind of means "overly apologetic, meek, timid, unwilling to ever voice anything that may be even mildly controversial", etc. It apparently originates from H.T. Webster "The Timid Soul"

"Alignment tax?"

If you want an AI to actually do things like answer your questions instead of just complete the question you have to nudge the model to do that which makes it stray from the "best" answer. There's a balance you need to strike between aligning it enough so that it gives answers humans actually want and not aligning too much to degrade the models performance.

People have an axe to grind with OpenAI because they purposely align their model to be boring as the intended audience is companies embedding it in their own products and try to fight the bias inherent in a model trained on the writings of the average chronically online Redditor. The frustration isn't unfounded, if you want to use the model for programming having to pay the 'alignment tax' because someone else is using the model for a customer service bot and can't match energy with an angry customer sucks.

alignment tax = Training needed to cancel bad behaviour, political incorrectness, dangerous results etc.

Yes, but you want alignment even if you don't want censorship. A very intelligent but unaligned model will be prone to doing useless things like "auto-complete" your question into a more elaborate version or responding with "just google it dumbass" and other forms of internet vitriol.

> Yes, but you want alignment even if you don't want censorship.

Instruction tuning helps a lot with this but what a lot of people mean is the refusal to do things. You get to chose how "aligned" it is, for some usecases like talking to customers you definitely want something very "safe" (won't start using slurs or something terrible). But for direct usage you generally never want it to refuse to do anything.

Checkout Anthropic on the extreme side - every iteration of Claude has gotten worse on the chatbot arena (elo based on humans blindly comparing responses).

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

The other replies are spot on but in general -- Alignment induces cost, no matter WHAT type of alignment, even 'neutral' versions like instruct for chatbots

The 'common' example that people reference is, for instance, making the model non-offensive means it has to spend.. something... on that directive, if you will. (which also led to absurdities such as refusing to help people with bash commands that involve having to ```kill``` a process...)

But even for the purpose of an instruct model (this is what peeves me off), making it answer questions and take instructions makes it *worse* at many creative tasks because you're constraining it's behaviour to Q&A -- though this is a long tangent...

Reply on news.ycombinator.com