What makes it left or right is how you otherwise spend the money saved from stopping subsidies - not the ending of subsidies itself.
If you reduce tax then it's right wing, if you use the money to help the poorer in society deal with the short term price shocks that would result from spiking energy costs - then it's left-wing.
But ending oil subsidies gets mentioned a lot on the left without that. Because ultimately it's also a subsidy for oil companies, and it adds to pollution.
I would suspect this is because it is most mainstream recommendation from economists on how to disincentivize fossil usage without destroying your society & economic stability in the interim. Not because it's "lefty".
The yellow vest protests in France were basically what happens when you try to do A without also doing B.
I don't know the situation in the US - and I suspect there are multiple factors - but you need to understand the goals, whether that be energy security, lower domestic prices, jobs or whatever.
ie the question isn't whether oil subsidies are good or not, in isolation - the question is what's the best method to achieve the goals.
( One goal you haven't mentioned is weaning the country off fossil fuels - which is probably best done via gradually rising prices ).