What is your opinion on flying for pleasure (i.e vacation)? We know that flying too is very damaging for the environment and creates terrible noise pollution. Would love to hear your thoughts on this!
What about taking the "scenic route" to get somewhere instead of the most direct route available? You still have a destination and purpose, rather than driving just for the sake of operating the vehicle, but you might drive a little further and take the car's emissions into a nicer area. What about driving a big, gas-hungry SUV or luxury vehicle to be comfortable instead of the most economical compact car that is feasible? What about people operating off-highway vehicles instead of hiking and using mule trains to get deep into the woods?
Starting with the same initial argument that it is wrong to operate a vehicle for pleasure because of its impact on others' health, why do we allow people to create filthy camp fires in the very locations where the most people will be exposed in national parks? Compared to a modern car, the fire is pumping out horrendous unregulated emissions...
For example, a driving tour of the French Alps would, I think, count as driving for pleasure, especially when you could take a coach and do the same trip. You can even pay to go on organised tours, such as http://petrolhead.tours.
To not be completely facetious though, it's going to be hard if not impossible for you to understand someone's passion for something, if you don't have any concept of enjoyment for it yourself.
A great many people find enjoyment in driving, for she sake of the activity itself. There are inherent risks and environmental downsides to it, for sure, but the same could be said for many things.
Your comment is however, the sort of thing that the grandparent comment was about - this attitude is expressed a lot on DN, and while this is an assumption on my part, it makes me feel that these people (such as yourself with your comment) must live in an environment where you haven't had the chance (or necessity) to use vehicles in the capacity that gives you an appreciation of the leisure aspect to them.
Even if that's not the case, however, it does pay to at least recognise that while you're strongly against something you can at least be objective in how you express your opinion.
BTW I think cars are super fun, I'm actively involved in motorsport, and I can completely understand when people aren't interested in it.