I can’t keep up with the millions of llm variants coming out each day. So far it’s Mixtral / Mistral that have caught my attention.
One of my biggest pet peeves with the LLM Benchmark culture is that in practice, marginal increases don't matter and are overall subjective. But using a strong model (like ChatGPT is still) is a good starting point.
And I agree that the leaderboards don't currently reflect the quantities of interest typically needed in practice.
That is very, very hard to do in an objective manner, as the current LLM benchmark gaming demonstrates.
Sure, you can deploy a smaller model to production to get real-world user data and feedback, but a) deploying a suboptimal model can give a bad first impression and b) the quality is still subjective and requires other metrics to be analyzed. Looking at prediction probabilities only really helps if you have a single correct output token, which isn't what LLM benchmarks test for.
Hopefully in 2024 we can get at least one of the benchmarks to move to assessing non-parametric/distribution-free uncertainty for selective classification, reflecting more recent CS/Stats advances that should be used in practice. Working on it.
The fine tunes (e.g. dolphin-mixtral, dolphin2.2-mistral, etc) may be good at coding or whatever else they specialize in, but they seem to sacrifice in other areas that result in 'hallucinating' where Mixtral/Mistral wouldn't - perhaps as a result of 'score chasing' on benchmarks like has been mentioned already.
Depends on your own specific needs & use cases whether this is important or not.
In my experience, Phind also seems fairly nice and is free for now, at least when the Cloudflare protection in front of it doesn't fuss over me using a VPN: https://www.phind.com
Aside from that, I just paid for GitHub Copilot (https://github.com/features/copilot), albeit JetBrains are also coming out with their own product (https://www.jetbrains.com/ai/).
It's nice that there are options that are basically "plug and play", both free and paid ones, if you don't feel like tinkering with the open source models, or don't have the hardware to run them with good performance (such as me only having an RX 580 GPU).