Hacker News Reader

Comment by joshstrange | Hacker News Reader

It's insane to me how fast this is moving. I jumped through a bunch of hoops 2-3 days ago to get this running on my M1 Mac's GPU and now it's way easier. I imagine we will have a nice GUI (I'm aware of the web-ui, I haven't set it up yet) packaged in an mac .app by the end of next week. Really cool stuff.

View on HN

I hope this kickstarts some kind of M1 migration. There are so many ML projects I'd like to try, but they all depend on CUDA.

Yep, I was just thinking the same thing. M1/M2 appears to be a huge untapped resource for ML stuff as this proves. I maxed out my MBP Max and this is probably the first time I'm actually fully using the GPU cores and it's pretty freaking cool. Creating landscapes or fictional characters (think D&D) is already super fun, I look forward to playing with img2img some more as well.

The performance gap to the top-end Nvidia cards will get much larger as they release new cards later this year, though.

Maybe, but I can buy a Mac, you just order one from Apple.

It's not clear if the shortages will happen with this new release as they did last time. Ethereum mining is going away and not as many people are stuck at home because of Covid. On the other hand, the performance increase looks to be substantial, increasing the demand.

An RTX 3090ti with 24GB of VRAM is widely available now that the crypto markets have crashed for $1150 or so. They were $2500 a year ago if you could find them.

A twist on the above comment: I _already own_ an M2 Mac, but I'm never gonna buy a high-end GPU to play around with this sort of tech. If the things people (who aren't gamers, crypto miners, or ML researchers) already own can be useful for some hobby-level work in the space, we'll see a lot more work and experimentation in the space. Its super exciting stuff.

To be fair now many people have gaming PC. Perhaps more than who have their own M1 Pro/Max.

It’s different kinds of people. I know more people with an apple mx computer than with a gaming computer (people have a console to game or nothing at all).

Perhaps, you think? ^^

The gaming PC market is huge. Have a look at https://www.businesswire.com/news/home/20210329005150/en/Glo... to get some numbers. There is a list of shipments in a year. Apple sells a lot of units, but not nearly enough to match the accumulated household supplies of gaming PCs - in how much, 2 years, while gaming PCs and laptops are still being sold?

Don't take that comment personally please, but this "perhaps" is a perfect example of being in a complete Apple bubble. It's so far from reality it is frankly unfathomable.

Arguably the cost effective solution is to use cloud services, since we're talking just a few seconds difference (or you might be lucky like one HN reader who got allocated an A100 today.)

But to play devil's advocate there are clear strengths available to the different platforms. PCs can readily upgrade into high end GPUs, but the compromise is that this becomes a requirement as basic GPUs don't feature enough VRAM and CPU-only mode is woeful.

On the mac side of things, the GPU is not going to be the latest and greatest, but the M-series features unified memory, so a relatively normal M-series mac is going to have the necessary hardware to load the models. Not the fastest (but still fast), and ready to go. (Also as it stands the M-series can offer additional pathways to optimisation.)

> Arguably the cost effective solution is to use cloud services

And running it on a fresh setup might help with the ‘works on my machine’ type of bugs that are being reported.

Same here. My M1 Max's GPUs were basically idling until this came along!

Do they depend on CUDA, or are they just much better tuned for NVIDIA cards? I thought the whole ML ecosystem was based on training models and then running them on frameworks, where model was sorta like data and the framework handles the hardware? (albeit with models that can be tweaked to run more efficiently on different hardware) (I don't really know the ecosystem so it is definitely possible that they are more closely tied together than I thought).

From my experience the bigger frameworks may have support for non-CUDA devices (that is not just the CPU fallback) but many smaller libraries and models will not, and will only have a CUDA kernel for some specialized operation.

I encounter this all the time in computer vision models.

The latter. The major frameworks, at least, can be run in CPU-only mode, with a hardware abstraction layer for other devices (like CUDA-capable cards, TPUs etc). So practically it means you need an Nvidia GPU to get anywhere in a reasonable amount of time, but if you're not super dependent on latency (for inference) then CPU is an option. In principle, CPUs can run much bigger model inputs (at the expense of even more latency) because RAM is an order of magnitude more available typically.

I was thinking (as someone who knows nothing about this really) that the Apple chips might be interesting because, while they obviously don't have the GPGPU grunt to compete with NVIDIA, they might have a more practical memory:compute ratio... depending on the application of course.

Is there any blocker to have VRAM swap (on RAM or SSD)? It would make processing much slower, but it should be better than nothing (cause OOM) or alternatively run on CPU (more slower).

Not sure. I suspect the issue would be lots of memory transfer between the GPU and the CPU, because downstream layers usually need previous layer outputs. It would probably depend on the receptive field of the network? Also on how expensive memory transfer is, maybe it's worth it in some cases. But there's no reason why you couldn't run say the first big layers on the CPU and then treat deeper layers (which may take a smaller input) as a separate network to run on the GPU. I suppose you want the largest subgraph in your model that can fit in available VRAM. Certainly the Coral/EdgeTPU will dispatch unsupported operations to the CPU but that affects all ops beyond that point in the computation graph.

It's not really better to move from one closed eco system to another. We should collectively agree to strengthen more open platforms, shouldn't we?

I'd rather see something more platform agnostic. I'm sad OpenCL isn't a bigger success.

OpenCL still works amazingly well on all platforms (e.g. two commercial programs of mine), it's just that everyone keeps saying it's dead and refusing to use it :(

Gods, yes. I've been trying to get various kinds of deepfake-related projects to work on M1 (or even just on Mac—the Intel Macs haven't come with Nvidia cards for years now) for some time now, as we're trying to generate video stimuli that present different people doing the same things, or the same person doing several different things, for psych research.

It's been an exercise in frustration.

Just yesterday I read another comment on HN saying we will have to wait another decade before being able train it in someone "basement"( https://news.ycombinator.com/item?id=32658941 ). I made a bookmark for myself ( https://datum.alwaysdata.net/?explorer_view=quest&quest_id=q... ) to look for data that help estimate when it will be feasible to run Stable Diffusion "at home". I guess it's already outdated!

Umm training is not the same as running it.

To run stable diffusion at home you have to download the model file, which took the equivalent of tens of thousands of hours spread across cloud provided GPUs.

If the model file just vanished from everyone's hard drive one day, and cloud providers installed heuristics to detect and ban image dataset training, retraining the model file would actually take decades for any consumer, even an enthusiast with a dozen powerful GPUs. The image dataset alone is 240TB.

You forget how much mark-up cloud providers charge.

I trained StyleGAN 2 from scratch using 8x 3090s at home and it took 3 months. It's fine.

240TB is small fish, my homelab is a petabyte and I consider it small.

It would take you way longer to train something like GPT-3 with such a setup.

Just to build my intuition, how long do you think it would take you to train Stabl eDiffusion in your homelab if you dedicated it to that task ? 10 years ? 20 years ? What about GPT-3 ?

Reply on news.ycombinator.com