This is a bit of surprise to me as well. That being said. The dataset is a tough one. I am ESL, but some of the accented examples I have a hard time understanding. Also the recordings are not near field and there is sometime some background noise.
We used entire librispeech as part of our training set.
Until you publish Librispeech WER for your model, it's pointless to discuss any advances in efficiency. I have no clue if your results on CommonVoice are good or bad.