Then you have the fact that transport, and hence fuel, affects the price of most goods, so fears of a recession (decrease in purchasing power) are not unfounded.
Also consider that, while justified as addressing a "tragedy of the commons scenario", rarely do such tax dollars go towards mitigating the externalities in question. The tax would sting less if it was being spent building renewable energy rather, than initiatives that are charitably described as contentious and perhaps more accurately described as pandering. If we could avoid the graft and self-dealing usually associated with federal spending, that would likely cut down on the complaints as well.