Two of my grandparents drove everywhere for decades, retired, still drove everywhere, then had huge trouble with their joints etc. When my grandmother lost her license (worsening eyesight), she was very isolated.
My other grandparents had nice cars, but also made good use of their pensioner's discount/free transport pass. My grandma would take the train across a large English city to visit her sister or friends. She's in her late 80s, and still very mobile, although no longer independent.
I prefer a ban, as it's fairer to everyone. There are externalities that are very difficult to price (pollution, accident risk), and just being rich doesn't give people extra rights to poison the air and kill/injure people. It also ensures rich people use public transport, so they have some incentive to make sure the service is good.