Running PyTorch on the M1 GPU | Hacker News Reader

Running PyTorch on the M1 GPU

sebastianraschka.com

92 pointsmy12342 comments

https://news.ycombinator.com/item?id=31424048

Accelerated PyTorch Training on M1 Mac (pytorch.org)

436 points by tgymnich 2 days ago 143 comments

MPS (Metal backend for PyTorch) is pretty poor integrated right now most of my custom models are crashing and pure Swift + MPSGraph versions is working 3-10x faster then PyTorch. So I'm pretty sure there is A LOT of optimizing and bug fixing before we can even consider PyTorch on apple devices (and this is ofc. 0.1 or smth like that so maybe we'll get something working fine in next few years).

Also it'll be cool to compare Tensorflow vs PyTorch on metal device :)

Also, even for ops that do not crash, it often returns garbage. I built PyTorch from git this morning:

    >>> torch.arange(10, device="mps")
    tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='mps:0')
    >>> torch.ones(10, device="mps").type(torch.int32)
    tensor([1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216,
            1065353216, 1065353216, 1065353216, 1065353216], device='mps:0',
           dtype=torch.int32)

Many bugs will probably be squashed over the coming weeks, so I am still very exited about MPS support. I have done some preliminary benchmarks with a spaCy transformer model and the speedup was 2.55x on an M1 Pro. Which is quite nice because transformers were already very fast on M1 Macs, thanks to the AMX units (PyTorch links against Accelerate, so the AMX units are used for matrix multiplication). If they can squeeze more performance out of M1 GPUs in the future, it will be a very nice speedup.

Weeks :? Looking at every other project from Apple that try to integrate with some opensource lib it'll be months/years if they didn't drop support for it before that :P

Did you get only 2.55x on BERT/other transformer vs CPU version?

I think in this case the PyTorch team also involved. They have already fixed some annoying bugs in the last two days (like matrix multiplication often failing because of buffer size mismatches).

As for BERT inference performance, I am not sure what kind of speedups you are expecting. The M1 Pro gives me 2.6 TFLOPs in single precision matrix multiplication of 768x768 matrices. M1 Pro GPU performance is supposed to be 5.3 TFLOPS (not sure, I haven’t benchmarked it).

Ahh nvm I was thinking about m1 max (my brain is damaged by the m1 naming and i didnt saw information that you are using m1 pro gpu)

Right, the Max should make a much bigger difference, since it has the same number of AMX units as the Pro, but double the GPU cores.

What ML are people actually wanting to do on M1 laptops?

I was under the impression that outside toy problems, you can't really get decent results unless you have a TPU or fleet of GPU's for training. If you just train on a laptop you're shooting yourself in the foot compared to someone who gets the same job done 10x faster in the cloud (usually with free credits).

Obviously inference is still possible on a laptop for many models... but not many 'hacker' tasks involve only inference.

I’m not using an M1 machine yet, but I think is extremely useful to be able to do some small scale experiments for testing and debugging purposes locally before starting a full-scale training job in the cloud.

Further, in general, I’m really charmed by the potential of having unified memory; the idea you can test some batch training iterations with a batch size that fills almost all 128MB is a unique capability!

Last, a large amount of unified memory allows to do inference and “prompt engineering” with very large models, locally. E.g. Using GPT-J like models (6B parameters).

What ML are people actually wanting to do on M1 laptops?

(1) development - quickly train smaller networks locally to validate the code. Then offload to a big GPU machine. (2) Smaller convolutional networks and RNNs may be below SOTA, but can still provide a good performance-accuracy trade-off. You could already train these networks pretty fast on an M1 thanks to AMX, but his makes training such networks even faster.

>I was under the impression that outside toy problems, you can't really get decent results unless you have a TPU or fleet of GPU's for training.

This is most certainly not the case. It's impossible to achieve SOTA on most benchmarks but there's a lot more ML and data analysis done on smaller datasets where you don't particularly need a fleet of GPUs.

The whole Diffusion text to image thing which is hugely popular right now doesn't involve any training. Same for text generation where you download pre-trained models. I'm using a laptop with a 3080 and getting much faster results than I typically get from a paid Colab account (not Pro mind you).

Which repo/tool are you using for local text to image diffusion if you don't mind sharing? :)

Not at all. I discovered that Visions of Chaos (originally developed for fractals etc) has extensive support for installing and running ML models

https://softology.pro/tutorials/tensorflow/tensorflow.htm

It handles (most) of the installation issues and gives you a GUI. You can however bypass the GUI. Every time you run a command the console shows you what the command line looks like so copy/paste into a terminal and away you go.

There's plenty of training related stuff that 10 TFlops is sufficient for. Fine tuning models for example is a great use case for a high ram yet relatively low TFlops environment.

I don't know how much you've done with running ml jobs on cloud boxes but it can be very annoying to work with them when you are actively iterating. Especially if you are doing a lot of work outside the cloud box environment. Even turn key consumer products like Colab can fall over if you accidentally press "Back" in your browser... I once lost many hours of work because I inadvertently swiped left :)

Apple will continue improving their performance in this space and once jobs that run for hours can be run overnight... I suspect people will often choose to run overnight simply for the ergonomics benefit.

Totally, performance for actual useful training etc. is at least a couple of hardware generations away. This is super useful however for people like me who like to use an m1 Mac as their daily driver and don’t always want to have to fire up a remote rig just to sketch a little. Also being able to do some local inference can be great for when building a web GUI, etc.

So absolutely not mature, but having PyTorch working better also on macs is only positive imho.

I've fine tuned transformers on CPUs before for weird reasons. Sometimes the computer you have is what you make do with.

Ideally fine-tuning foundation models. Training them costs several millions, so out of scope no matter which GPU you have.

Can someone explain to me why M1 Ultra doesn’t scale ratably compared to RTX? It seems it should perform at about 1/2 of 3090. Is it because Apple locked down what you can do with the GPU?

If the tensor cores are used, the 3090 can hit 300Tflops. I don’t think the M1 can even come close to that (21Tflops for the ultra)

Makes sense so it’s basically 7x more just in terms of raw power. I was wrongly under the impression that Apple was limiting ML capacity.

1/2? 1/2?!?

make it at least 1/10

Too bad there's a requirement for metal. I had tensorflow running on a raspberry pi 3 a few years ago doing inference for a chat bot.

Why wouldn't GPU support for the M1 GPU require Metal? You should be able to run PyTorch on the CPU of a Pi?

Metal is an unnecessary complication.

How so? NVIDIA GPUs have their own APIs, AMD does, Direct3D is its own. The Apple platform acceleration APIs are Metal. The MS acceleration APIs are all generally Direct3D, etc.

The fact the you want Apple to support other platforms doesn't mean that they're wrong for not doing so. They same arguments to require Apple to support your preferred API also applies to MS, NVIDIA, AMD, etc and the same arguments to not do that apply them all as well.

That other companies use shitty proprietary code to lock out competition doesn't make Apples attempt to do the same any better. It compounds the error and wasted effort. In fact Apple's is the worst of the lot you mention because at least you can use alternatives with the others.

Apple’s is the result of their experience promoting OpenGL and OpenCL. They found GL and CL both remained resolutely in the past, and people also seemed resolutely opposed to using them.

So GL and CL had to make a compelling argument for why apple should continue to invest in the technology, and even now they still aren’t able to show a single compelling argument for them being better than any alternative.

Personally I would have rather a single open API, but the open API did not appear to be super interested in competing with the other platforms. The end result is the open platforms turn into a pile of expense, licensing, and engineering for behind the curve technology and API.

They don't have to develop it. They just have to not prevent it from existing. If theirs is genuinely better, it will get used. But Apple is too scared to openly compete so they lock down and artificially limit competition.

Anyone can implement the opencl and OpenGL APIs on top of metal. Apple isn’t preventing that.

But native OpenGL requires apple writing OpenGL implementations for their hardware. It requires apple spending engineering time implementing misfeatures from OpenGL and opencl in order to support APIs that are only every considered the backup API.

Vulkan. If they'd open their tech up, none of this stuff would be a problem.

What about Vulkan? They’d need to implement the same crummy APIs and semantics (that is a bunch of kernel and similar work) for an API that NVIDIA and AMD were both uninterested in.

And again, anyone could have made a Vulkan emulation layer on metal.

Then what API are you going to use on macOS?

You're not permitted to use anything else, thus: too bad it has a requirement for metal.

Nice, does not look super fast, but a great start and looks very usable for development or smaller tasks

(a great start especially considering the missing nvidia GPUs on intel macbooks)

Does the M1 support openCL/SYCL on macos?

OpenCL 1.2* with very few extensions, pretty much stuck in the past. Don’t even think about using unified memory from there.

SYCL: due to the above, not on the GPU. Just on the CPU.

* x86_64 apps also get to see the OpenCL CPU backend in addition to the GPU one. However, arm64 apps only see the GPU.

Apple is an irrelevant company at this point

On the contrary, I believe MacBooks have continued to eat general market share since they switched to Apple Silicon, and they are by far the best built UNIX-based laptop you can get, and have by far the best battery life. There’s a reason so many devs (and scientists, like myself) use them.

They're the fourth largest laptop vendor, making 9% market share, up from 7% the year prior [1]

Apple is #2 world wide for phones at 18% [2]

Apple is #1 for phones in the US with 51% share [3]

Apple is #2 for phones in the EU at 23% share [4]

Apple's laptop seem to wipe the floor in power usage, while not getting anyway more in gaming land than any prior generation [5]

Apple's revenue is the 3rd highest in the world which also doesn't seem particularly irrelevant.

You can hate Apple all you want, but going "I don't like apple, therefore the company is irrelevant" is a dumbass approach. Hell what will android get without finding out what's in iOS?

[1] https://9to5mac.com/2022/04/11/mac-market-bucks-trend-with-c...

[2] https://www.canalys.com/newsroom/global-smartphone-market-Q1...

[3] https://www.imore.com/apple-dominates-us-smartphone-market-5... (This is iMore so take with a grain of salt)

[4] https://www.appleworld.today/2022/02/28/apples-iphone-has-23...

[5] https://www.pcmag.com/news/intel-core-i9-vs-apple-m1-max-whi...

Maybe for those that buy Apple as shinny UNIX instead of supporting Linux OEMs, for others there are plenty of market opportunities.

Apple aggressively pushed openCL when it was new, and NVIDIA and AMD aggressively pushed their platform specific CUDA APIs, so all anyone did was complain about apple using openCL when no vendors supported it.

Apple had (runtime and ABI) stable openCL built into macOS and iOS by default and on desktop devs kept requiring you install NVIDIAs or AMDs CUDA systems, and on iOS everyone found that openCL was just as clunky to use as openGL.

Apple made proprietary APIs that were more usable and better interacted with the hardware, and everyone just used those instead, until Apple deprecated openCL and openGL as they merely represent a pile of engineering work and support that is essentially unused.

Reply on news.ycombinator.com