Hacker News Reader

You'll probably need 9 or more. 4090s have 24GB each.

Oops, I read 48 somewhere but that's wrong. Thanks.

a6ks however

I wonder if AutoAWQ works out of the box, given no architectural changes (?). That would be most straightforward together with vLLM for serving.

If an implementation had NVidia's Heterogeneous Memory Management implemented, then 192 GB RAM DDR5 + GPU VRAM would seem to be close.

Two 128gb Mac studios networked via thunderbolt 4?

This is actually a promising endeavor. Id love to see someone try that.

There's already at least one project that attempts this: