Deep Learning’s Diminishing Returns (2021)
spectrum.ieee.org
spectrum.ieee.org
If you used today's level of computation with 2012 era models, then you wouldn't have the error rates of today's models, you would have much higher, much worse errors.
The single biggest bottleneck in deep learning is not computation, it's the ingenuity used to devise new structures and new optimizations that allow for scaling.
For a given structure, with given techniques, you can throw more computation at a problem to decrease error rate, but the gains scale poorly with cost. Cost improvements only come with new techniques and structures.
In other words, in order for us to buckle the trend we would need to start innovating in ways that are unlike the ways that got us from AlexNet to here.
However, the obvious retort here is that if you train a model that is good at generalizing, then you don't need train more models! A show of hands, who has used GPT or an open LLM vs who has trained one would yield a vast disparity. If you don't need generalization, you don't need huge models. Small models are efficient over narrow domains and don't require vast compute/energy resources.
Secondarily, it's a self-solving issue. Energy isn't cheap, and GPUs aren't cheap. If you're going to burn 10's of thousands of dollars in energy costs, you should probably have a decent reason to do it. But those reasons are quickly diminishing as things that have already been _done_.
Third, overparameterized models are becoming less of an issue during inference with efficient quantization techniques. Distillation, though harder, is another option. Again, you do can these things one time after training.
Believe me, that has almost nothing to do with "only needing one model" and everything to do with the compute required being absurd.
> Small models are efficient over narrow domains and don't require vast compute/energy resources.
The "best" small models are still typically born from the "best" overall models (however large) using a student/teacher paradigm. This is active research. As such, it is typically encouraged to go for "the big one", as you can distill it to far better small models than if you were to train the small models from scratch first.
> If you're going to burn 10's of thousands of dollars in energy costs, you should probably have a decent reason to do it. But those reasons are quickly diminishing as things that have already been _done_.
Again, this is _not_ the reason researchers don't train their own GPT-4's. They _absolutely_ would if they could raise that much money. The notion that researchers have everything they ever could have wanted from OpenAI's API-based walled garden approach is patently absurd. If they would release the weights, then finetuning and such could occur and you might have a point.
To be clear, I'm not defending the article's main point. You just happened to have used some strange arguments against it (imo).
The best argument is a sibling comment which points out that new architectures will be discovered that are more parameter efficient and run better on the hardware available. The burgeoning field of architecture search may mean we even have the ability to get the models themselves to help speed up this process of finding better ML architectures. It's always difficult to know when we have hit a wall of sorts however, and it may be that the transformer (and similar approaches) can only be refined so much. Time will tell.
If DL costs are unsustainable, it means those other things (SAP, web hosting, and all the ridiculous stuff people waste cycles on) are even more unsustainable, and should be addressed first.
That fundamentally misunderstands the point. If it costs me $XXX amounts of dollars to compute the job I'm trying to replace, which only actually costs $X dollar, then it's not financially viable to use the model to replace the job. It makes no difference what the global percentage of computation that makes up.
Just like not optimizing the 1% bottleneck in your code when there's a 50% bottleneck- focus on the big players where the cheap, easy wins are first.
The paper explicitly says: """Important work by scholars at the University of Massachusetts Amherst allows us to understand the economic cost and carbon emissions implied by this computational burden. The answers are grim: Training such a model would cost US $100 billion and would produce as much carbon emissions as New York City does in a month. And if we estimate the computational burden of a 1 percent error rate, the results are considerably worse."""
(that paper was debunked by people who build and operate extremely large scale cloud machine learning systems).
Remember, Google already built and runs its TPU fleet continuously and the machines are almost always busy at at least 50% of their capacity, meaning the money is already being spent and the carbon (which is much smaller than their estimates) being spewed.
As for the rest of the paper, their arguments about the need to increase parameters to get better results are fairly simplistic, and filled with misleading information/untrue statements.
Basically, everything they claim in the article has been shown to be empirically not true. And the community is already grappling with the "too many parameters approach". The authors would have served the community far better by writing a less critical paper that focused on the opportunities to identify good approaches to parameter and compute reduction that don't affect performance. In looking at the main author's area of research, it looks like he is more of a economist/public policy wonk than DL expert, which I think negatively affects the quality of the paper.
Articles are often talking about something that won't end up mattering. But if you can't actually address the point the article is making on its own terms, you're going to struggle to refute it.
When I refute a paper I start with the weakest points; if I convince my audience to ignore the paper on those terms, I don't spend time refuting the main point (perhaps this is not a great technique, but it is fairly efficient). I didn't even really address their main arguments because after reading the paper, I was just aghast at some of the things that say.
Let's give an example like this: """The first part is true of all statistical models: To improve performance by a factor of k, at least k**2 more data points must be used to train the model."""
I guess that's true of nearly all modern statistical models, at least small ones with simple model functions and fairly small number of data points. But I don't think that most advanced deep learning experts think in those terms; modern DLs do not behave anything like classical statistical models. I think those experts see increasing the data as providing an opportunity for overparameterized systems to generalize in ways we don't undertand and don't follow normal statistical rules. Modern DL systems are more like complex systems with emergent properties, than statistical models as understand by modern statisticians.
Here's another example: they sort of ignore the fact that we only need to train a small number of really big models and then all other models are fine-tuned from that. To get a world-class tardigrade detector, I took mobilenet, gave it a few hundred extra examples, trained for less than an hour on my home GPU (a 3080Ti), and then re-used that model for millions of predictions on my microscope. I didn't have to retrain any model from or use absurd amounts of extra data. I took advantage of all the work the original model trainign did to discover basis functions that can compactly encode the difference between tardigrade, algae, and dirt. I see a direct linear increase in my model's performance as I move to linearly larger models, and I need to add linear number of images to train more classes. We can reasonable expect this to be true of a wide range of models.
Similarly, for people doing molecular dynamics- the big CPU waster before DL- many parts of MD can now be approximated with DLs that are cheaper to run than MD, using models that were just trained once.
What about AlphaFold? Even if it cost DeepMind $100M in training time (and probably more in salaries), it has already generated some results that simply couldn't be produced without their technology- it didn't even exist! What they demonstrated, quite convincingly, is that algorithmic improvements could extract far more information from fairly cheap and low quality sequence data, compared to expensive structural data. So instead of extremely expensive MD sims or whatever to predict structure, you just run this model. My friends in pharma research (I work in pharma) are delighted with the results.
In short, I think the author's economic model is naive, I think his udnerstanding of improvement in DL is naive, and he undercounts the value of having a small number of huge models which are trained on O(nlogn), not n**2 data sets.
And I think that in the next decade it's likely either Google, Meta, or Microsoft will be actively training multi-modal models that basically include the sum of all publicly available, unencumbered data to produce networks that can move smoothly between video, audio, text, do logical reasoning, everything required to produce a virtual human being that could fool even experts, and probably even exceed human performance in science and mathematics in an impactful way. So what if they spend $100B to get there. That's just two years of NIH's budget.
Deep Learning's Diminishing Returns - https://news.ycombinator.com/item?id=28646256 - Sept 2021 (84 comments)
Each picture has more than one pixel, so for the network to "remember" the images, we need more parameters than there are pictures.
Better results for general purpose tasks are likely to find willing buyers at prices orders of magnitude higher than even the largest language models of today. In most cases, the best alternative is human labor, and for more latency-sensitive cases, there is no alternative to a language model available at any price.
https://en.wikipedia.org/wiki/Carbon_offsets_and_credits#Con...