These arguments are always phrased as if every guy who's having trouble with condoms is an idiot. I think that's counterproductive.
I'm a "grower". The size difference is so significant that any condom that still feels comfortable when erect will easily slip off when flaccid. Smaller condoms are actually painful because of the ring at the base.