First, without cars, keeping commute times reasonable requires that you either live very close to your job (which I admit is ideal anyway) or that you have a very, very robust intercity commuter transportation system. Something we're not even close to having.
Second, we can't move everyone and their brother to city center, or even within the city limits. A very large fraction of workers live outside city limits. Moving all these people with in a distance of rail stations that makes building lots of them cost effective would be very difficult from a city planning perspective.
Third, our economy has been operating on the notion (for quite sometime) that home ownership is ideal. Since people have a lot of their personal wealth wrapped up in their home, asking them to move is not really fair either. This affects "non-car users" too when that move is going to raise the prices of housing where the non-car users live.
Fourth, allowing people to move about easily from any city to any city means that there is liquidity in the job market. This is both for car owners and not. The fact that you can buy a car and go somewhere a train or bus doesn't means you can take a new job when the current one doesn't meet your needs.
There are also other costs that are saved (both monetary based and time based ones) for non-car users. Having fewer people in the city limits means that your wait times for services is lower than it could be. It's easier to get in and out of the city on a weekend.
That said, I'd be willing to pay a more direct tax for the convenience of owning a car if it made people feel better.