Hacker News Reader

Comment by bitexploder | Hacker News Reader

No one on the planet could run a frontier model. The amount of VRAM you need would bankrupt a normal person.

I suspect most people don't have enough storage space to even download a frontier model.

This comment makes no sense. The original comment wasn't specifically to do with the average person hosting at home.

There are already: - hundreds of western providers hosting open models, including large ones such as Kimi K3 which is near Opus/Sol sizes.

- people hosting large models such as Kimi K3 at home. Check on reddit, there are people with 20x DGX Spark setups etc.

I was being dramatic, but no normal person is running 20x DGX spark. That is 100K of slow VRAM. Not sure it makes a lot of sense TBH.

20? I have 1 and it seems they can only be configured up to 4.

I was replying to parent suggesting you buy 20 DGX to run a multi terabyte model as a normal human / hobbyist.

all the enterprise clients which the two frontier companies rely on have the money to buy the hardware and run a frontier model themselves, it just becomes a nobrainer to buy your own hardware if you have 10-20b token output per week

Well... given that I have codex reporting 1b tokens consumed for my couple-days long session, I suspect 10-20b is not that hard to reach for a company of 10 devs. It would be interesting to know what is "rental fee" for a model like Astra or Sol 6.1, and how much hardware they actually need - not just gpu, but all of it?

To be comfortable you need 300 to 400 terabytes of storage. You need the GPUs and the cooling for them. They don't need a lot more. You put models in memory and you do math really fast and make tokens. You need some CPU and RAM infrastructure for like prompt caches and things like that, which does add up, but the dominant cost is by far the GPUs and VRAM. Everything else is a pretty easy step down to afford. You probably need to have multiple different models, right? Like if you want an Astra plus a Sol plus, you know, like a Luna-class model, say, you probably need like 10 terabytes for Astra, probably 3 to 4 terabytes for Sol, and 1 to 2 for Luna, maybe less, maybe one or half a billion even for Luna. That's RAM. You need So all total you need 15 to 16 terabytes of active VRAM to keep those models resident. That's not counting what you want to keep in cash and whatever hosting constraints these models have that aren't public. That much VRAM is five to eight million dollars right now.

Plus what models are you running? The biggest models you can run are like in the terabyte range or Qwen or GLM. Kimi K3 is there, but GLM 5.3 is very competitive. Using voice dictation, hands hurt, sorry for any errors. You still want many terabytes of RAM to run say multiple GLM+DeepSeek V4.1 for some actual concurrency.

We have had this discussion for a decade regarding cloud hosting. There are definitely workloads and AI inference that you want to do in-house. Hosting these GPUs and models and setting up the interconnect and balancing it is really hard right now. It's not traditional infrastructure. It is exotic. Typical enterprise is at such a disadvantage to try and host this completely themselves. And models are constantly updating. And what models are you going to run? Six months ago, things that probably cost you a billion tokens probably cost you 50 to 100 million now, given how much more efficient models have gotten, or at least they cost that much. I don't know. It's just not a given.

Reply on news.ycombinator.com