This is largely cherry-picking revisionism. I saw CGA on the displays of the time at the time with the games of the time, and can confirm it was mostly terrible. EGA couldn't come fast enough.
EGA was definitely a massive step up, no question. But people also miss the point of CGA when they look as sharp screenshots rendered on a pixel perfect LCD display.