Understanding, using, and finetuning Gemma
lightning.ai
lightning.ai
I tried the open source release yesterday. I started with the input string "hello" and it responded "I am a new user to this forum and I am looking for 100000000000000..." with zeros repeating forever.
Ok, cool I guess. Looks like I'll be sticking with GPT-4.
Many models are being released now, which is good to keep OpenAI on their toes and not mess up, but, truth be told, I've yet to see _any_ OSS model that I can run on my machine being as good as ChatGPT 3 (not 3.5, not 4, but the original one from when everyone went crazy).
My hopes for consumer hardware ChatGPT-3.5 within 2024 probably lie with what Meta will keep building upon.
Google was great, once. Now, they're a mere bystander in the larger scheme of things. I think that's a good thing. Everything in the world is cyclic and ephemeral and Google enjoyed their time while it lasted, but, newer and better things are and will, keep on coming.
PS: Completely unrelated, but, gmail is now the only Google product I actively use. I don't, genuinely, remember the last time I did a Google Search... When I need to do my own digging I use Phind these days.
Times are changing and that's great for tech and future generations joining the field and workforce!
I still couldn't run Mixtral 8x7B on an M1 Macbook Pro with 32Gb ram, so maybe I am indeed doing it wrong? Or are there better quantized versions available now or..?
But GGUF Mixtral should fit in 32GB... just not with the full 32K context. Long context is very memory intense in llama.cpp, at least until they fully implement flash attention and a quantized cache.
It depends on your machine I guess, but IMO there's definitely OSS models out there that rival the original ChatGPT offering for certain use cases(dolphin mixtral comes to mind). Having a model with RAG capability is going to make a huge difference in the quality of the answer, as well.
They don't have the augmentations of being a service, but generally they are smarter, have a bigger context and (perhaps most importantly) are truly unbound.
I am on a single 3090 desktop, for reference. Admittedly, this is much more expensive now than it was a few months ago, with the insane prices used 3090s are going for now.
On a Macbook M2 I get ~10/12t/sec which is a tiny tad bit too slow for continued/ daily use, but if I think its worthy I might invest on a more powerful machine soon-ish!
Miqu 70B is slow (less than 10 tok/sec, I think) because I have to split it with llama.cpp. I only use it for short context questions where I need a bit more intelligence.
And for reference, this is a SFF desktop! It's no Macbook, but still small enough (10L and flat) for me to fly with in carry on.
Also, Gemma is a +9B model. I think it's not okay that Google compared it with Mistral and Llama 2 (7B) models.
Google also took llama.cpp and used it in one of their Github repos without giving credit. Again, not cool.
All this hype seems to be backed by Google to boost their models whereas in practice, the models are not that good.
Google also made a big claim about Gemini 1.5 1M context window, but at the end of their article they said they'll limit it to 128K. So all that 1M flex was for nothing?
Not to mention their absurd approach in alignment in image creation.
(Edit: I'm wrong)
Gemma 7B is a 9B model. The name is a lie. Then they really played games with Gemma 2B as well.
I don't get how Google can be this incompetent and far behind everyone else. They have amazing people and the kinds of resources that almost no one else does but somehow need to resort to faking demos, blatant lies about model sizes, etc.
Google used to be the place everyone wanted to go. Someone at Google AI needs to be fired so they can start being productive again.
That's the danger of too much abstraction, it's easy to have big gaps in one's understanding of what's really going on.
Ohhh so that explains why I couldn't load it on my RTX 4090, while other 7B models load just fine!
Are you talking about gemma.cpp? Then no, they didn't.
The claim is correct but not related to gemma
> Run LLMs locally on Cloud Workstations. Uses:
> Quantized models from [Huggingface]
> llama-cpp-python's webserver
But sure, the blog post doesn't mention it.
Yes that's correct. It's 9.3B parameters if you count the embedding layer and final projection layer separately. However, since they used weight tying, the adjusted count is 8.5B as discussed in the article.
do you even LLM?
- Local models are pretty easy to de-censor, if thats what you mean.
- ...Yeah, it should not be labeled as a 7B. Its sort of 7B class.
- The repo mentions they use the llama-cpp-python server
- 1M context brute forced across TPUs is insanely expensive, I can see why Google reigned it in.
But overall your message is not wrong. Google is hyping Gemma a ton when its... Well, not very remarkable. And they could have certainly made something niche and interesting, like a long context 8.5B model, a specialized model, a vastly more multilingual model, something to differentiate it from Mistral 7B 0.2
About 50% of the shots, I get a sentence and a half of beautiful poetry, then a codeswitch into kanji, and then ral ral ral ral ral 膳 ral 杯 ral ral
Until I kill the process. Not every time, but way more often than the other llamas (which is basically never, these days).
I think they underestimated the impact of training on bulleted lists. It seems to love those!
They said it was inspired by llama
>This is inspired by vertically-integrated model implementations such as ggml, llama.c, and llama.rs.
He meant this:
https://cloud.google.com/blog/products/application-developme...
They say it's because they're not counting embedding parameters[0]. Although apparently even with the embedding parameters subtracted it still rounds to 8B not 7B. From what understand, rounding to the nearest B is the standard. Seems slightly disingenuous to call it 7B, but not a big deal IMO since I don't hear anyone saying this model is outperforming popular OSS 7Bs.