I think you'd have to be pulling a pretty long bow to argue that fossil fuel extraction and use has no environmental externalities - this paper for example (http://insight.jbs.cam.ac.uk/2015/measuring-fossil-fuel-hidd...) shows that for 2008-2012 "the implicit subsidy exceeded post-tax profits (averaged over five years) at all 20 of the largest oil, gas and coal producing companies studied".
We're already (globally) subsidising the fossil fuel industry, I think it's only fair to ask for some of that money back - it's not like they would 'unpollute' the sky and sea if they could anyway.
While they may be the foundation of the modern era, fossil fuels are also a dirty relic that should be left in the past. Leaving aside the environment impact I don't think I'd have to argue so hard that they have a not-so-great legacy when it comes to huge injections of capital into the middle east and it's concomitant propping up of dictators and despotic dynasties.