Hacker News Reader

Comment by scott_s | Hacker News Reader

Big Sur is twice as fast as our previous generation, which means we can train twice as fast and explore networks twice as large.

Can a machine learning expert comment on this claim? There are many domains in computing where doubling your compute power does not halve your execution time, or allow you to double the size of your input. (For example, your algorithms are worse than O(n), or you have hardware communication bottlenecks.)

View on HN

Author here - in this case we are talking mostly about ~halving the time of a single minibatch in training a variety of standard neural networks. That type of training tends to be very hard to distribute. The servers also have 8 gpus in a single host, which is an additional performance boost and isn't included in the 2x above.

I'm curious why this claim (distributed minibatch is hard) continues to be repeated. Am I misreading the state-of-the-art? There was a paper at Interspeech this year [0] where Amazon showed excellent scaling (54X speed up when distributed across 80 nodes) without loss of accuracy.

[0] http://www.nikkostrom.com/publications/interspeech2015/strom...

This is not really true. ML applications in general do not scale linearly. There is the systems (scaling) overhead, and then there are the algorithm payoffs, which start to diminish depending on the algorithm. If they went from 4 to 8 GPUs, the only thing they can guarantee to double is how fast they burn power.

These are machine learning experts making the claim?

Machine learning is usually embarrassingly parallel.

If I was reviewing a paper with bits on machine learning (presumably written by machine learning experts), I would ask machine learning experts I know to verify the claims I was not familiar with.

I do know that a lot of ML algorithms become, at the low-level, matrix operations. Matrix operations are typically not embarrassingly parallel. Now, I can imagine a bunch of independent matrix operations, but whether you'd call the entire problem "embarrassingly parallel" would depend on how coarse the independent bits are.

In many cases you can train multiple instances of a neural net simultaneously in increments, average the weights together, and repeat. Making the underlying compute more efficient is an interesting problem.

Reply on news.ycombinator.com