I used to think like that, but the city I live in has regular no car days that made me change my mind.
The first and biggest pollution is the noise. The ambient car noise has a huge effect on stress level and sleep quality. When you remove the car from the city, it's almost at silent as the countryside.
The second is the danger in the street. Your brain registers those big fast moving cars as potential danger to avoid and raises your stress level and makes you want to move away from it.
The third one is the space the car take. When you remove the cars you have the opportunity to either make things easier to walk to, or to add parks and greenery. When you remove cars, modern cities are surprisingly empty, much less dense than a village.
Then there's the air pollution.