Hacker News Reader

Comment by ojosilva | Hacker News Reader

Apple mentions TensorFlow explicitly in the ongoing presentation due to the new 16-core "Neural Engine" embedded in the M1 chip. Now that's an angle I did not expect on this release. Sounds exciting!

Edit: just to clarify, the Neural Engine itself is not really "new":

> The A11 also includes dedicated neural network hardware that Apple calls a "Neural Engine". This neural network hardware can perform up to 600 billion operations per second and is used for Face ID, Animoji and other machine learning tasks.[9] The neural engine allows Apple to implement neural network and machine learning in a more energy-efficient manner than using either the main CPU or the GPU.[14][15] However, third party apps cannot use the Neural Engine, leading to similar neural network performance to older iPhones.

Source: https://en.wikipedia.org/wiki/Apple_A11#Neural_Engine

View on HN

I can't see how something that tiny can compete in any meaningful way with a giant nVidia type card for training. I'd imagine it's more for running models that have been trained already, like all the stuff they mentioned with Final Cut.

Yeah I would imagine it's intended for similar use-cases as they use for iOS - for instance image/voice/video processing using ML models, and maybe for playing around with training, but it's not going to compete with a discreet GPU for heavy-duty training tasks

For an 18-hour battery life computer (Macbook Air) that now doesn't even have a fan, it's for a complete different market segment from where nvidia cards dwell.

Not all NN models are behometh BERTs, U-Nets or ResNets. Person detection, keyword spotting, anomaly detection... there are lots of smaller neural nets that can be accelerated by a wide range of hardware.

Can anyone who knows about machine learning hardware comment on how much faster dedicated hardware is as opposed to, say, a vulkan compute shader?

That depends entirely on the hardware of both the ML accelerator and the GPU in question, as well as model architecture, -data and -size.

Unfortunately Apple was very vague when they described the method that yielded the claimed "9x faster ML" performance.

They compared the results using an "Action Classification Model" (size? data types? dataset- and batch size?) between an 8-core i7 and their M1 SoC. It isn't clear whether they're referring to training or inference and if it took place on the CPU or the SoC's iGPU and no GPU was mentioned anywhere either.

So until an independent 3rd party review is available, your question cannot be answered. 9x with dedicated hardware over a thermally- and power constrained CPU is no surprise, though.

Even the notoriously weak previous generation Intel SoCs could deliver up to 7.73x improvement when using the iGPU [1] with certain models. As you can see in the source, some models don't even benefit from GPU acceleration (at least as far as Intel's previous gen SoCs are concerned).

In the end, Apple's hardware isn't magic (even if they will say otherwise;) and more power will translate into higher performance so their SoC will be inferior to high-power GPUs running compute shaders.

[1] https://software.intel.com/content/www/us/en/develop/article...

On the NVidia A100, the standard FP32 performance is 20 TFLOPs, but if you use the tensor cores and all the ML features available then it peaks out at 300+ TFLOPs. Not exactly your question, but a simple reference point.

Now the accelerator in the M1 is only 11 TFLOPs. So it’s definitely not trying to compete as an accelerator for training.

Isn't it better to rent a cloud with as many GPUs as necessary for a time needed to train the model? I don't know state of things in ML.

Curious to hear responses to this too..

For productionizing/training massive models, yes.

But in the development phase, when you are testing on a smaller corpus of data, to make sure your code works, the on-laptop dedicated chip could expedite the development process.

I agree with the parent poster that it's probably more about inference, not training.

If ML developers can assume that consumer machines (at least "proper consumer machines, like those made by Apple") will have support to do small-scale ML calculations efficiently, then that enables including various ML-based thingies in random consumer apps.

Not necessarily.

It can be surprisingly cost-effective to invest a few $k in a hefty machine(s) with some high-end GPU's to train with due to the exceedingly hefty price of cloud GPU compute. The money invested up-front in the machine(s) pays itself off in (approximately) a couple of months.

The "neural" chips in these machines are for accelerating inference. I.e. you already have a trained model, you quantise and shrink it, export it to ONNX or whatever Apple's CoreML requires, ship it to the client, and then it runs extra-fast, with relatively small power draw on the client machine due to the dedicated/specialised hardware.

Cloud GPU instances are very expensive. If you get consumer GPUs not only do you save money, you can sell them afterwards for 50% of the purchasing price.

Sure, inferencing doesn't need floating point instructions, so NVIDIA will stay the only real solution for desktop/laptop based model training for a long time.

It is just not meant to be a training device so comparing with data Center or developer GPUs is useless. Faster inference for end-users is what is mentioned by Apple and the only use case where this hardware makes sense.

It's kind of sad that AMD spent years to get mediocre TensorFlow support and Apple walks in with this. It really shows how huge Apple is.

But they needed to, MacBooks are simply no option if you want to train models. I dont expect crazy performance but would be great if MacBooks would be an option again for prototyping / model development at least

Who mentioned training? Most of these chips are only any good for inference. A wonderful symphony with the Apple computing mantra, of course.

Totally agree.

Besides, can the neural engine be used to speed up other tasks?

Could this also be remedied by Apple supporting Nvidia GPUs again? And then you plug in a beefy eGPU?

I think it is interesting that Apple is at Nvidias mercy again with the Nvidia ARM deal. Hopefully they get their shit together

In fairness, it's been possible to convert a TensorFlow model to a CoreML model for a while, and in April TensorFlow Lite added a CoreML delegate to run models on the Neural Engine.

https://blog.tensorflow.org/2020/04/tensorflow-lite-core-ml-...

So don't think of it as Apple walked right in with this so much as Apple has been shipping the neural engine for years and now they're finally making it available on macOS.

I think this is for inference not learning, even though they use the term machine learning. They seem to just mean running models based on machine learning approaches.

Tensorflow includes stuff for inference.

My thinking was along the same line as yours, but the way apple framed it seems to suggest that the M1 accelerates model training and not just inference. Here's the actual quote "It also makes Mac Mini a great machine for developers, scientist and engineers. Utilizing deep learning technologies like tensorflow ... which are now accelerated by M1". It should be pretty straightforward to test this though: installing tensorflow-gpu on a mac mini and seeing the result. I suspect, TF's latest branch should also indicate which GPUs are supported. Curious to hear more thoughts on this.

They do not have any hardware combination which can actually support even modest GPU intensive training sadly, so much touting running models instead of training.

Yeah, I was confused at that implication. I don't think these are designed for training!

(If they are or can be, I'm interested)

> (If they are or can be, I’m interested)

Exactly. Currently I am training my models using Google Colab and then exporting the model to run on my MBP. Would be interesting if I could do it locally

Another interesting thing is that ( if this is for training ) this will become the only accelerated version of Tensorflow for macOS as: - No CUDA drivers for latest macOS - AMD ROCHm only supports Linux runtime

I’m hoping the M1 can be used for prototyping with small data sets, then final training on Google Colab with complete data sets.

Is this just opinion? Maybe they are designed ALSO for training. I wonder if this things can replace nVidia graphic cards on training? The neural core has a LARGE area on the chip design similar to the GPU area.

Yes, need to see more strong evidence that the new MBP's can handle large amounts of ML Training using TF or CreateML so we don't have to get NVIDIA machines/laptops.

Would love to get GPU accelerated learning in Lobe (http://www.lobe.ai).

Reply on news.ycombinator.com