So... they're saying it used about $100 worth of electricity.
[ https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 ]
[ https://www.statista.com/statistics/190680/us-industrial-con... ]
Wasn't Google prioritizing green energy for their datacenters?
Google is the largest corporate purchaser of renewable energy in the world
The whole point of having AI ethicists is to identify current indicators of potential future ethical problems so that they can be considered in guiding the direction of development, so that you minimize acute ethical crisis.
You realize the shear scale of co2 output the office the engineers who wrote the model, drove to work, education, etc produced.
I’m actually suprised just how small of a co2 output it was.
Sure, the issue is that the scale of models is increasing by orders of magnitude in a fairly short span of years, as well as the range of applications rapidly expanding. It doesn't take long for that kind of growth to go from a trivial issue to catastrophic one, and it's the exact kind of risk you have ethicists in a field to call out while it still is trivial so that some of the energy of people doing technical development gets directed to mitigate the risk of it ever reaching the catastrophic stage.
Capitalism actually mitigates the risk. Companies won’t pay 1 million in R&D to replace 100k of salary.
In an ideal world, nobody would get hung up on details and everybody would understand that there is a lot of nuance when comparing things. In practice, if Google published a paper which quoted the Strubell paper without the caveats, I can see headlines about how inefficient and bad for the environment Google Translate is. PR would get busy and obtain corrections or follow-up articles to clarify things, but those rarely get the same attention. And it's still extra work that could have been avoided, which in itself is bad optics ("do you folks even review the stuff you send out for publication?").
I'm all for reducing emissions and improving efficiency, but I find the premise a bit of a stretch.
Yes, large and wealthy organizations have a big advantage, but that applies to pretty much anything they do, not just language models. Inefficiencies are bad, but if you ask a number of people why fix them, I think they'd mention financial cost and wasted time well before looking at it as an issue of ethics and fairness.
Reducing costs for language models is already a great idea across all fronts. So is reducing inequalities. It's linking the two that sounds like a strained argument to me. Suppose someone makes models ten times smaller next week. Will marginalized communities' lives improve soon? There must be more. I haven't seen the paper, so I'm curious how it is all framed.
An alternative motivation could be to launder accountability so that when the acute ethical crisis /does/ come, you can throw your hands in the air and say "See? Look at how much resources we poured into this and it still happened! At least we tried!"
Gasoline is a hydrocarbon, but the mass of hydrogen is neglegible compared to that of carbon, so it's not too wrong to say that this mass of carbon came from the same mass of gasoline, 250 kg. The density of gasoline is around 1 kg/l, so we are talking about 250 liters of fuel. The typical tank of a compact car holds about 50 l.
It is like someone complaining that google map uses x, amount of energy, yet the old alternative (printing maps, and getting lost) was much worse CO2 wise.
It seems obvious that this would be something an ethicist would be interested in researching to figure out what the impact will be and start discussions about how to offset it.
The total amount of electricity being used so far for training models does not yet move the needle in a significant way, but the point of the paper is that it's currently growing in an unbounded exponential fashion. And you know how exponential functions work.
Models like BERT aren't just trained once either when they are developed, but trained again with different domains, different parameters, different tasks in some cases. There is also fine-tuning (more frequent, less carbon intendive), so these are real environmental problems, and others have pointed them out.
Now about the billion cars on the road and the 40% of the world’s electricity being generated from coal.
How much more of a problem are a billion cars and 40% of our electricity being generated from coal?
We’ve squandered decades ignoring the big problems and now people want to run into the weeds with a thousand little problems, that individually don’t amount to much.
CO2 impact for stuff like this - we probably have bigger fish to fry.