This is a bit of surprise to me as well. That being said. The dataset is a tough one. I am ESL, but some of the accented examples I have a hard time understanding. Also the recordings are not near field and there is sometime some background noise.
What is the WER of your model on Librispeech?
We used entire librispeech as part of our training set.
Until you publish Librispeech WER for your model, it's pointless to discuss any advances in efficiency. I have no clue if your results on CommonVoice are good or bad.
In my experience DeepSpeech with the default Mozilla model + 5-gram is extremely sensitive to background noise. Fiddling with the hyperparameters, e.g. upweighting the language model, helps a little.
Thanks. Do you use the default parameters and tune from there?