Hacker News Reader

> is ultimately a silly and misguided proposal

No its not.

Its analogous to quantifying an F test with watts.

Nothing wrong with that. I can give a billion row dataset to a dozen people. John assumes normality, samples 100 rows, runs a linear regression, gets 70% R^2, under 1 second on a 1980s era computer. Mary doesn't assume normality, runs a glm via irwls, gets better explanatory power than John, still on 1980s computer, though she takes 5 seconds instead of 1. Then James comes along & runs a decision tree & is twice as good as Mary, but he now needs a second on a 1990s PC. Tony uses a random forest & Baker wants a 128 layer neural net. And so on & so forth, until we end up with burning enough energy that would otherwise power a village, to overfit some dataset that excel at some completely artificial leaderboard metric.

At some point a grown up comes in & says you don't need to light up your bedroom with industrial stadium lighting if you need to do your homework, a 40w table lamp will suffice. Hell half the third world uses a candle & they are performing quite well, if not better. That's really all this is.

The marginal gains in performance aren't justified compared to the amount of energy you end up expending. Half these models are brittle & have zero shelf life, ppl regularly throw away stuff they wrote just a year ago, so to what raison d'etre this pursuit ?

The cure isn't proscribing "green ai", it's adding a carbon tax that embeds the cost of the carbon externality into the cost of power in general, then letting the free market make the decision.

> to overfit some dataset that excel at some completely artificial leaderboard metric.

This comment exemplifies the myopic narrowminded focus on costs and being dismissive of the benefits of greater resource investments which I am criticizing here. Said 'completely artificial' tasks like ImageNet actually turn out to transfer really well to the real world and are why AI is becoming so useful, and also spurred the research into how to create efficient NNs via distillation and compression which still cannot be done without training the large models first. You need to invest if you want efficiency. Performance precedes it, it doesn't follow it. As the programming saying goes: get it correct, then get it fast.

> The marginal gains in performance aren't justified compared to the amount of energy you end up expending.

Come on, we’ve been through this before. At least for the past 10 years if not much longer, performance improvements have not actually improved performance, they’ve improved developer productivity.

If you create a language that’s half the speed, but developers can program twice as fast in it, it makes sense to use it. Processing power is much cheaper than a $150k+ developer. Also, being first to market can yield outsized gains on competitors.

Money is optimizing this already as the highest invest is computing resources.

>This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or "price tag" of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods.

[emphasis mine]

They suggest reporting cost metrics next to benefit metrics so readers can judge the tradeoff for themselves. There's no proposal to ignore the benefits.

And, as I already pointed out, none of their suggests cost metrics are even close to a total cost, and many of them are seriously misleading, and they use them in misleading rhetorical ways (as I noted about their bogus estimates of CO2 emissions). Everything about this paper, from the name 'green AI' on down, is framed to put sole attention on a narrow arbitrary section of costs of political relevance, at the cost of a holistic evaluation. A single vague clause about 'other measures' doesn't change that, no matter how you italicize it. Motte-and-bailey.

It is useful, though, to compare results of projects with different budgets. There are papers tackling the same problem with 4 orders of magnitude difference in computing. Including an objective metric in the results would be worthwhile.

In this case, the metric I want is either TFLOPs (for a commercially available architecture) or kWh (for some in-house machine) rather than tons of CO2, which depends on their power source.

I disagree with some of this. The main thrust of their paper is correct, IMO: cost of training (and inference) needs to be factored into the comparison when comparing models. You can't optimize what you don't measure, and currently many SOTA results have the training regimes which take a ton of compute and aren't really doable in a research context if you're not Google and can't just use racks full of idle TPUs to do your stuff for free in the trough of the diurnal cycle while customers aren't using them.

Cost of training is currently ignored so much, some of the authors don't even mention it in their papers, yet it has a direct relationship with things like researcher productivity and reproducibility of results by mere mortals.

I wouldn't call this "myopically focusing on the cost of training". I'd call this "acknowledging cost of training as a valid metric worthy of publishing and optimization". I hope everyone can agree with this. Once you start optimizing this in addition to your mAP or BLEU or top1 accuracy, the results could be multiplicative.

I suspect this will ultimately come down to math: both having architectures which are amenable to efficient gradient propagation and having better optimizers.

There is no reason given to think that anyone is ignoring the total cost of training and not factoring this in when comparing models.

Do you think Google is funding all this research and just forgetting to consider the cost of training BERT? Do you think any researchers are ignoring the cost of renting GPUs or their university overhead? AI researchers talk all the time about costs! They're positively obsessed with it, in a way you don't see in any other area. (When was the last time you saw someone snicker about a CRISPR paper, 'yeah, nice results, but you know those reagents cost thousands of dollars, right?' Or covered in said paper?)

It's quite a double standard. Someone wants to spend >$20 billion on ITER (which doesn't work), you never hear about it; someone sketches out that the most inefficient way to train a neural net model (using on-demand cloud GPUs) costs $1m, suddenly it's a huge deal, a boondoggle, researchers should be ashamed of themselves, it's a threat to research, and we need 'green AI'.

I worked at Google for many years. Google has so much idle compute at any given moment, they tend not to care _at all_ about you using it. It's not a problem for a Google employee to just grab a few thousand cores on a whim and do whatever they want with those cores, and many people do use this freedom. No permission is required if your tasks are preemptible (which by design they mostly are, at Google). I'm not sure if the same applies to TPUs (I left before they became prevalent) but I'm sure for the right subset of people the right permissions could be procured. Most workloads follow the diurnal cycle, so at any given time anywhere between 20 and 80% of capacity is sitting idle. That's far more than you need to do most kinds of deep learning research. That's why you see work from them that would cost you $100K+ to replicate end-to-end (e.g. neural architecture search). At Google you get those resources for basically free, so it's not a huge issue to just spin up a rack full of TPUs. That's just how things are done over there.

I don't get your opposition to introducing approximate exaflops spent in training as an additional metric by which to judge "goodness" of a model. It is well known that some models are much easier to train than others. It's also known that some optimizers converge faster than others. All this stuff directly affects reproducibility of research. We're getting to the point where some of the papers aren't reproducible unless you're at Google or DeepMind or OpenAI or Microsoft Research. I think you'll agree that this is pretty terrible.

Between two models with approximately the same validation accuracy metrics and inference timings wouldn't you choose the one that's more economical to train? I know I would. Why aren't these metrics even mentioned most of the time? Beats me. Even better if the model requires less _data_ to train, but that seems to be less tractable than just estimating and surfacing the cost of training.

From what I understand from other Googlers, there are internal markets designed to track and allocate compute resources; this is necessary because of their expense, and because of the low marginal value of many possible uses.

> I worked at Google for many years. Google has so much idle compute at any given moment, they tend not to care _at all_ about you using it. It's not a problem for a Google employee to just grab a few thousand cores on a whim and do whatever they want with those cores, and many people do use this freedom.

And it works out for them, does it not? Because the uses are useful on net, even if some of them seem speculative or silly or inefficient. If they weren't, if people were just constantly burning resources to do absolutely nothing, presumably things would change and those resources would be tracked by the markets too or other measures taken. Let Google as a whole decide where to optimize, end to end - don't push it onto individual researchers and guilt-trip them into worrying about whether they are 'green AI' or evil people who hate the planet.

> I think you'll agree that this is pretty terrible.

I would not. I think it's awesome that machine learning is seeing the tiniest crumbs of the resources other fields have (still grossly disproportionate to its potential) invested in it, and we are no longer in a field where all results are reproducible on a $200 gaming GPU run overnight, and this is delivering big gains in performance and surprising nontrivial demonstrations of scalability (which would not be true of, say, IBM Watson). Again, as I said, imagine applying this sort of standard to any other field of science. 'You can't discover the Higgs boson using just 250 watts and a $200 interferometer you bought off eBay, I think you'll agree that this is pretty terrible.' Reproducibility doesn't mean 'should be reproduced, frequently, by anyone, at arbitrarily small budgets'.

> Between two models with approximately the same validation accuracy metrics and inference timings wouldn't you choose the one that's more economical to train?

No, I'd choose the one with the lowest total cost for my purposes. Do you program only in assembler to make sure you write programs with only the most economical runtime costs? What a narrow way of seeing costs... I'm often much more concerned with which one is easier to modify, or to scale, or to reuse for transfer learning, or which runs acceptably for users. The model which is 'more economical to train' is only better on one of many axis, and when someone else like OA or DM pays for it, it is a sunk cost of zero relevance to me. (How much does it matter to me how long it took GPT-2 to train? Not very much! Because I download the trained model and finetune it for other things.)

> Why aren't these metrics even mentioned most of the time?

They typically are. Most DL papers will mention roughly the GPU/TPU count and wallclock time, or you can guess from the training graph or baselines. And that's really all you need to know as you evaluate trainability as one of many cost variables; stuff like CO2 emissions or flops are not.

> There is nothing special about training an AI; it is simply another thing to spend resources on

If the goal is actual intelligence, as the 'I' in the name appears to suggest, then making resource consumption in the form of energy/compute a central part of the equation makes a fair deal of sense. It's arguably long overdue and might actually push research towards methods that actually do increase the capacity of agents to learn rather than just praying for faster chips and more data.

The benefit here can be twofold in that more inclusivity increases the pace at which the field advances by including more researchers, but also just gives researchers an incentive to not treat compute/data as unlimited moving closer towards figuring out how the brain actually learns.