Carbon emissions arguments tend to ignore the value of what's being done as well. BERT and other transformers were meaningful experiments that were valuable in furthering a major research direction and enabling more effective consumer and business applications. In that sense, it's like any other company doing R&D - of course energy will be used and of course there will be some inefficiencies.

I think it's quite misleading to compare the energy usage of an industry-wide research effort to individual consumption. The graphs look bad - "wow, 626,000 lbs! that's 284 metric tons of CO2! a plane flight is way less!" - but there's a fundamental difference between "progress on a problem being worked on by thousands of highly-paid researchers" and "I bought a car".

Meanwhile, the worst power plants are generating on the order of 10+ million tons of CO2 every year. There are at least a dozen of these in the US alone. Car factories are emitting hundreds of thousands of tons of CO2 (Tesla is somewhere around 150,000 tons a year, apparently, and it's designed to be efficient). Perhaps activism around CO2 emissions in ML training might be better focused on improving the efficiency of those things instead, seeing as a 1% improvement would outweigh the entirety of the NLP model training industry. It's certainly good to keep in mind the energy costs of training in case things balloon out of control, but right now the costs relative to the results seem small and not worth highlighting as some forgotten sin.