The energy requirements for growing cotton being largely the sun, and a resource which makes up most of the universe (yes, I know water is several steps from hydrogen). ((And the tangentials like the fertilizers, dieslel etc.))

Compared to the energy requirements of petroleum based products: Incredible density and pressure acting on sufficient organic matter(thousands of times greater than growing cotton), on millenium time scales, then requiring several incredibly expensive and commoditization refining/distribution practices, and the end product is then ceremoniously burned off as poison or discarded into the aether.

It is 100 percent the "gotcha OP thought it is". However I do see/realize the fallacy you're pointing out in that the lesser of two evils is still evil.