Was it via gemma.cpp or some other library? I've seen a few people note that gemma performance via gemma.cpp is much better than llama.cpp, possible that the non-google implementations are still not quite right?
One thing I do suspect people are running into is sampling issues. Gemma probably doesn't like llama defaults with its 256K vocab.
Many Chinese llms have a similar "default sampling" issue.
But our testing was done with zero temperature and constrained single token responses, so that shouldnt be an issue.