The distance between you and the impact of your increased fuel consumption or environmental impact is massive compared to the distance between you and the impact of stopping/starting at every single stoplight. For drivers of a car, it's almost imperceptible. How do you quantify the difference? How much more are you going to pay at the pump? For the cyclist, you feel the impact immediately, paid in sweat.
I'm not saying it's ok or that it's a good justification. I'm just saying that there's a clear difference in the way people will weigh one against the other under different circumstances.