Hacker News Reader

Comment by etaioinshrdlu | Hacker News Reader

What type of hardware would you need to run it?

A cluster of many $8000+ gpus. You're looking at around 350GB of vram, so 30 12gb gpus - a 3090 will cost around $1800, so $54k on the gpus, probably another $15k in power, cooling, and infrastructure, $5k in network, and probably another $20k in other costs to bootstrap it.

Or wait 10 years, if gpu capacity scales with Moore's law, consumer hardware should be able to run a ~400GB model locally.

Almost no one does this on prem. What would this cost on AWS?

This is not true. On prem is extremely common for things like this because after ~6 months you'll have paid more in cloud costs than it would have cost to purchase the GPUs. And you don't need to purchase new GPUs every 6 months.

AWS would cost $50-100k/mo for something comparable.

> so 30 12gb gpus - a 3090 will cost around $1800

3090 has 24Gb, thus 15 GPUs X $1800 = $27,000 in GPUs

Can 3090 GPUs share their memory with one another to fit such a large model? Or is the enterprise grade hardware required?

Yes, two 3090s ($1.7k each) can be connected via NVlink with common 48Gb of memory pool.

Two RTX A6000 ($4.5k each) can form 96Gb memory pool.

One could use $4.5k RTX A6000 48Gb instead. They can be joined in pairs of 96Gb common memory pool with NVlink. That’s 7x$4.5=$31.5k in GPUs to get 336Gb of memory. Or 8x$4.5=$36k in GPUs to get 384Gb of memory.

Add say $3k per GPU pair for surrounding computer (MB,CPU,RAM,PSU) 4x$3k=$12k.

$48k total budget.

Reply on news.ycombinator.com