Hacker News Reader

Comment by adventured | Hacker News Reader

There was a leak from Japan yesterday, prior to this release, and in that it was suggested 20gb for the largest model.

This text was part of the Stability Japan leak (the 20gb VRAM reference was dropped in the release today):

"Stages C and B will be released in two different models. Stage C uses parameters of 1B and 3.6B, and Stage B uses parameters of 700M and 1.5B. However, if you want to minimize your hardware needs, you can also use the 1B parameter version. In Stage B, both give great results, but 1.5 billion is better at reconstructing finer details. Thanks to Stable Cascade's modular approach, the expected amount of VRAM required for inference can be kept at around 20GB, but can be reduced even further by using smaller variations (as mentioned earlier, this (which may reduce the final output quality)."

View on HN

Thanks. I guess this means that fewer people will be able to use it on their own computer, but the improved efficiency makes it cheaper to run on servers with enough VRAM.

Maybe running stage C first, unloading it from VRAM, and then do B and A would make it fit in 12 or even 8 GB, but I wonder if the memory transfers would negate any time saving. Might still be worth it if it produces better images though.

Sequential model offloading isn’t too bad. It adds about a second or less to inference, assuming it still fits in main memory.

Sometimes I forget how fast modern computers are. PCIe v4 x16 has a transfer speed of 31.5 GB/s, so theoretically it should take less than 100 ms to transfer stage B and A. Maybe it's not so bad after all, it will be interesting to see what happens.

If you're serious about doing image gen locally you should be running a 24GB card anyway because honestly Nvidia's current generation 24GB is the sweet spot price to performance. 3080 ram is laughably the same as the 6 year old 1080Ti and 4080 ram is only slightly more at 16 and costs about 1.5 times the 3090 second hand.

Any speed benefits of the 4080 are gonna be worthless the second it has to cycle a model in and out of ram anyway vs the 3090 in image gen.

> because honestly Nvidia's current generation 24GB is the sweet spot price to performance

How is the halo product of a range the "sweet spot"?

I think nVidia are extremely exposed on this front. The RX 7900XTX is also 24GB and under half the price (In UK at least - £800 vs £1,700 for the 4090). It's difficult to get a performance comparison on compute tasks, but I think it's around 70-80% of the 4090 given what I can find. Even a 3090, if you can find one, is £1,500.

The software isn't as stable on AMD hardware, but it does work. I'm running a RX7600 - 8GB myself, and happily doing SDXL. The main problem is that exhausting VRAM causes instability. Exceed it by a lot, and everything is handled fine, but if it's marginal... problems ensue.

The AMD engineers are actively making the experience better, and it may not be long before it's a practical alternative. If/When that happens nVidia will need to slash their prices to sell anything in this sphere, which I can't really see themselves doing.

>How is the halo product of a range the "sweet spot"?

Because it’s actually a bargain second hand (got another for £650 last week buy it now eBay) and cheap for the benefit it offers for any professional who needs it.

3090 is the iPhone of AI, people should be ecstatic it even exists not complaining about it.

> because honestly Nvidia's current generation 24GB is the sweet spot price to performance

You're aware the 3090 is not the current generation? You can see why I would think you were talking about the 4090?

Think it's weirder you assumed I meant 4090 when it doesn't really offer enough benefit over the 3090 to justify it's cost, and you mentioned an incorrect price for the 3090 anyway so it's not like you weren't talking about 3090.

> If/When that happens nVidia will need to slash their prices to sell anything in this sphere

It's just as likely that AMD will raise prices to compensate.

You think they're going to say "Hey, compute became competitive but nothing else changed performance therefore... PRICE HIKE!"? They don't have the reputation to burn in this domain for that IMHO.

Granted you could see a supply/demand related increase from retailers if demand spiked, but that's the retailers capitalising.

The price hike would come proportionally with each new hardware generation as their software stack becomes competitive with CUDA. Since it will most likely take them multiple generations to catch up. It doesn't make sense for them to sell hardware with comparable capability at a deep discount. Plenty of companies will want to diversify hardware vendors just to have a bargaining chip against Nvidia.

If it worked I imagine large batching could make it worth the load/unload time cost.

Shouldn't be a reason you couldn't do a ton of Layer C work on different images, and then swap in Layer B.

Reply on news.ycombinator.com