Hacker News Reader

$50k would be the cost to run it un-quantized, 10k could get you for example 4 5090 system, that would run the 671b q4 model which is 90% as good, which was the OPs target

View on HN

which 671b quants can fit into 96GB VRAM? Everything I’m aware of needs hundreds at least (e.g. https://apxml.com/models/deepseek-r1-671b).

5090 is 32 GB so it's 128GB, not 96.

128 is still not 300. Something like 4x 6000 blackwell is the minimum to run any model that is going to feel anything like claude locally.

To my deep disappointment the economics are simply not there at the moment. Openrouter using only providers with zero data retention policies is probably the best option right now if you care about openness, privacy and vendor lock-in.

For local use and experimentation you don't need to match a top of the line model. In fact something that you train or rather fine-tune locally might be better for certain use cases.

If I was working with sensitive data I sure would only use on prem models.

Reply on news.ycombinator.com