AI training is outpacing Moore’s Law | Hacker News Reader

AI training is outpacing Moore’s Law

spectrum.ieee.org

77 pointspseudolus46 comments

The article is a bit lacking in details. Here are some changes in ML performance that are happening today and don't require increased transistor counts:

1. newer optimizations algorithms converge faster (Adam, adagrad)

2. lower-precision numerics can be used (TF32 is 19bit, bf16 is popular)

3. matrix multiplications are implemented in hardware (tensor cores, AMX)

4. neural networks are compiled directly (TVM, halide)

5. multi-device network topologies are tuned for training (e.g. nvlink improvements)

6. companies employ grad-student descent (hire a bunch of PhDs and task them with improving MLPerf results)

The last one is particularly effective.

I didn't really understand #6? Expand?

You can overfit your data by coming up w/ a model that is very specific to it. Consider an n dimensional data set where one variable is a boolean.

A researcher might notice that the True and False cases are different enough to warrant their own algorithms. Say they now fit a lower dimensional line to the True and False cases.

They've just performed gradient descent but the grad student was the gradient. It's a common joke in CS/ML departments.

(This is an obviously simple example, but the point is that hyperparameter tuning and algorithm selection _are a part of your algorithm_ and the data you're looking at while doing so is part of your training set)

> companies employ grad-student descent (hire a bunch of PhDs and task them with improving MLPerf results)

No joke, always cheaper than massive optimization tasks. I’ve managed a team working on automl for years and I can say with certainty, the cheapest option is always hiring interns (grad or bachelor students) or a hackathon

I'll throw in a seventh...

Neural architecture search is finding more efficient architectural building blocks which translate to better quality models with fewer overall parameters.

There's a lot of work going on right now to improve transformer architectures in particular; I think once the dust settles they will be massively more efficient than the current models. Efficient transformers often also introduce an inductive bias, which ends up improving model quality. (similar to replacing fully connected networks with convolutional networks; more efficient, but also carrying an inductive bias that improves model quality.)

We're also realising we don't need full transformers all the time. Token mixing might suffice, there are quite a few papers replacing the core of the transformer with something simpler.

>lower-precision numerics can be used

I wonder if it would be worth starting with low precision and then over time raise precision to zero in on a local min / max.

These large models are going to be the operating system of the future. When they are going to run efficiently on edge we are going to see an explosion of automation. Being so large they are not only more accurate, but also quicker learners. So less initial effort to start a project.

Google is teasing us with leaks about Pathways[1], but not releasing a paper or demo yet.

[1] https://blog.google/technology/ai/introducing-pathways-next-...

I call them multi-multi-models: multi-modality, multi-task, multi-language, multi-memory (besides the processing memory it can have search engine in the loop and episodic memory). Somebody's going to make it in the next couple of years.

The sad part is that we are all helping to train the futures AI but who will own it and make money on it?

You can have ownership in companies that do machine learning. It's called "shares". Creating your own business is an option too.

The complaint was that it's the people who own the equipment (or shares in it), and not the people who do the work, who are reaping the profit.

When the working class had this problem a couple centuries ago, it was solved by building guillotines. I hope we can do better this time.

To start an AI company you generally don't need much. A few thousands in GPUs and you are good to go. If AI researchers feel they are underpaid there is literally nothing stopping them from starting a company themselves.

that is absolutely not true. I'm not sure where you are getting your information, but read anything from an angel investor. most AI start ups fail because they go into huge debt getting correct data and training models, which end up bankrupting them in the process with nothing to show for it

If you need to collect date then you are not only an AI company. And in that case a large company is indeed needed.

You’re conflating two things. Feudalism is like slavery: taking credit for someone else’s labor. Automation removes the need for labor. Automation has almost universally led to quality of life improvements, not revolution.

I don't see any problem with it, stock market is open for everyone.

Nowadays startups often delay going to the stock market, and remain private while they're growing fast. It's only when they are growing more slowly and the VCs want to get out that they go public.

This way, the average investor usually does not reap outsized benefits from the ride.

> who will own it and make money on it?

The nice part about it is that everyone will have access to great models for direct use, fine-tuning and adaptation. But if people don't take this freedom and are content with simply consuming applications, then they depend on those who deploy the apps.

IMHO (as a AI researcher) 'multi-modality, multi-task, multi-language, multi-memory' may take longer than you think. Doing multi modal learning is quite hard due to lack of labeled data, a lot of tasks either don't offer clean plentiful datasets or require RL / human in the loop learning and not just supervised or self supervised learning, and on and on. Certainly a lot of progress will be made and partial automation is likely to speed up (as has happened with Copilot), but full automation seems to be a good while away.

Plug - I expand on this a bit in this blog post 'The Inherent Limitations of GPT-3' (which was on front page of HN a few weeks back and led to a good deal of discussion ) https://lastweekin.ai/p/the-inherent-limitations-of-gpt-3

I love this because it's vindicating Rich Sutton's essay titled "The Bitter Lesson".

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation."

And now we are beating out Moore's law!

Seems like a step towards singularity.

Do we even still have Moore's Law? All indications are that it died out a while ago.

Really?

http://www.transistorcount.com/

Slope seem a bit more gentle - but certainly not gone?

You're right about literal Moores Law, but its benefits are much diminished, of performance, power usage and cost. My understanding is the main problem is heat dissipation.

I've long predicted macrochips to solve this, brute force of huge quantities of silicon, but it hasn't really happened.

Graphics cards and neural accelerators have continued an exponential improvement in performance per watt dollar, which is the intuitive thing that people often mean when they say Moore's law.

If we hit a trillion transistors by 2030 without increasing chip size, I'll be shocked.

I guess it's inevitable that we'll reach a "hard" limit at some point, with exponential increase in density (electrons and photons have a "size"). But I don't know how that translates to current densities (how many atoms in a gate on the Apple m1 chips, and what is the theoretical minimum?).

If you're going strictly on number of transistors, I think we're still going up up and away. If you're talking about performance, which is what most folks really care about, we really are hitting thermal limits especially with consumer hardware.

AMD, Nvidia and intel are all hitting the 'heat wall' with their hardware just trying to cram more into smaller nodes and boost clocks. Nvidia's 4000 series cards are rumored to be 500W cards which is well into 'space heater' territory.

Transistor density/cost still helps with the inherently-parallel ML workloads.

I guess the next step is democratizing ML.

Right now we seem to be into a kind of "mainframe phase", where running big models is super expensive.

I wonder what kind of speedup we need for ML to be truly accessible for the average developer. 50x? 100x?

Define "accessible". Very modest hardware (sub $100 and certainly sub $1000) can train random forests very quickly. This is going to be close to SOTA on basically any classification task.

Random forests weren't even state of the art in 2010.

Are random forests new? no. SOTA in what? Image classification, obviously no. But they are still going to perform very well in the vast majority of business problems in classification.

> speedup we need for ML to be truly accessible for the average developer. 50x? 100x?

The data scale (and access) is often more important than the modelling, and that doesn't have a technological fix, or at least not a purely technological fix.

So Moore's Law is a constraint for average people as hardware costs a key factor in performance, is that right?

What I'm saying is the hardware is less of a problem than access to data, for most problems that would get you out of what you can reasonably do on a consumer accessible (cost) machine.

Unless we have widely available pre-trained models with generic knowledge that is easily transferable. Such a thing is obviously difficult, but it isn't impossible if we're talking about the future.

The trick for ML accessibility is minor finetuning the existing models that have been trained on these super training models so that users can get similar performance, and lately companies have been more open about releasing models open source.

Google is all over this -- there are at least two science-fiction level videos on YouTube of actual field experiments using ML, each with a very non-USA/third world person narrating.. the view count is in the low four digits both times

one of them is related to a paper in Nature this year on "democratizing ML" .. the MOSAIK project iir

edit- hilariously, Google Search wont let me look for MOSAIK instead correcting the spelling.. and when forced, many similarly named projects that are not it, are ahead.. the one referenced here is

    A generalizable and accessible approach to machine learning with global satellite imagery

If I'm reading the plot from the article right, the most impressive gains have happened in the first half of 2019. Since then, the progress has been notably slower, with year-on-year performance increases for some network architectures falling behind Moore's law already.

Yup, and we don’t currently have an architecture that can support models 35+ years from now.

Not yet at least.

Have we considered yet whether this is, to quote Stephen Diel, another handwavy technobabble nothingburger? I have very serious misgivings about all this what we currently call AI. Also, even if it is something, is it beneficial to all? Sure it can be beneficial to some corporates/VC but we didn't stop at the dawn of social media to think on the societal impact and we already know there's a lot of algorithmic bias being cooked here.

Reply on news.ycombinator.com