Reducing the Computational Cost of Deep Reinforcement Learning Research

It reads to me as a response to Gebru's paper that led to her getting fired. A paper on the externalities of training large models, like the environmental impacts due to the large amount of computation required: "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"

https://dl.acm.org/doi/10.1145/3442188.3445922

That would explain why such a mundane subject is in a blog post, it's focused at tech news to be all "we really do care about this thing" rather than academia.

So, it really sounds like we need to reduce our carbon emissions for electricity generation.

The windmills and solar are still on the way. Looks like the new goal in the US is 2035:

https://www.scientificamerican.com/article/bidens-infrastruc...

See this article [1] on the retracted paper. The article says:

"Gebru’s draft paper points out that the sheer resources required to build and sustain such large AI models means they tend to benefit wealthy organizations, while climate change hits marginalized communities hardest. “It is past time for researchers to prioritize energy efficiency and cost to reduce negative environmental impact and inequitable access to resources,” they write."

Meanwhile TFA concludes:

"We are simply urging researchers to consider smaller-scale environments as a valuable tool in their investigations, and reviewers to avoid dismissing empirical work that focuses on smaller-scale environments. By doing so, in addition to reducing the environmental impact of our experiments, we will get both a clearer picture of the research landscape and reduce the barriers for researchers from diverse and often underresourced communities, which can only help make our community and scientific advances stronger."

Looking more closely:

Gebru: "It is past time for researchers to prioritize energy efficiency and cost to reduce negative environmental impact and inequitable access to resources"

TFA: "We are simply urging researchers to consider smaller-scale environments... and ... to avoid dismissing empirical work that focuses on smaller-scale environments. ... in addition to reducing the environmental impact of our experiments, we will ...reduce the barriers for researchers from diverse and often underresourced communities"

So yes the similarity is striking. In fact its remarkable that google fired Gebru for questioning the retraction of the paper only to realize that these factors must be considered and then to publish nearly the same message six months later.

Having worked at Google, I think they wanted Gebru to sugar coat the message more. That’s what they’ve done in TFA. But in doing so they’ve also taken away a publishing opportunity from a talented black woman (who they just proved right) and I think that’s a shame.

[1] https://www.technologyreview.com/2020/12/04/1013294/google-a...

She said that they could discuss a possible resignation when she returned from her preplanned vacation. They said don't bother you don't work here anymore.

Threatening to resign is no more resigning than threatening to fire is actually firing.