Exactly. The amount of energy spent on heating and cooling is huge. Then it's transportation and agriculture. Then somewhere far behind is the whole Internet and efficient electronic devices.
I use the same argument when it comes to bitcoin. That stuff is so tiny in our profiler. Sane thing to do is to optimize the slowest functions.