I find these CO2 arguments from model training extremely misleading. If this is what you care about, why don't you actually try to make electricity cleaner. Realistic, that's a much more effective way to address this tiny issue, together with the grand total of 50% of humanity's CO2 emissions on top. To me it always seems like virtue signalling, showing that you "care", while not having to actually make any realistic progress. Go build nuclear power plants, or at least prevent them from shutting down, if you want you're electricity to be at 12 g CO2 / kWh rather than 500 from natural gas or 800 from oil (you can also move your servers to France or Ontario). Do not focus on tiny, 10% efficiency gains in NLP. That's a marginal gain within a miniature cause to start with.