Was not very impressed with the chat either.
So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.
One thing I do suspect people are running into is sampling issues. Gemma probably doesn't like llama defaults with its 256K vocab.
Many Chinese llms have a similar "default sampling" issue.
But our testing was done with zero temperature and constrained single token responses, so that shouldnt be an issue.
It could have been long context? Or a little bigger, to fill the relative gap in the 13B-30B area? Even if the model itself was mediocre (which you can't know until after training), it would have been more interesting.
I can't share the eval, but it's pretty simple: it asks a question about some data, and is restricted to only answer yes/no (based on the output logits and suggested in the prompt). It's called with 0 temperature and only 1 output token, so sampling shouldn't be an issue.