Hacker News Reader

> Will all these GPUs be used for inference once a SOTA model's training checkpoint/batch is done?

I have no real data to back this up, but that has always been my assumption.

Claude says that K3 can be assumed to have required 10-100M GPU hours. If you have 100k GPUs that would mean like 6 weeks of training. 100k GPU's can serve 3-30 trillion tokens of K3 per day. Google apparently serves ≈100 trillion per day [0].

The big labs probably want to have capacity to fairly quickly train / post train different SOTA models continuously + being able to serve peak inference demand in valuable markets (US daytime?).

[0]: https://blog.google/innovation-and-ai/sundar-pichai-io-2026

Your just missing many things and so have an incorrect picture of the situation - k3 is not on par with fable or astra. Closed models remain far better than best open weights at least today. - compute is not just used for training, more and more is inf - even in training you don’t do one run, you do many. Final run is a small portion of total compute.

The world is extremely compute constrained currently, like extremely.

This is evidenced by the prices on every large-model capable device/node increasing significantly over the last year. Demand is far exceeding supply.

That, or they are not operating profitable... If demand eats up actual pricing remains to be seen.

Ah, they're not profitable for sure. For example, OpenAI is losing tens of billions a year.

K3 feels a little under-trained. There's a huge gap between how much it knows when you talk to it, vs how well it performs on coding benchmarks. I'm quietly hopeful for some big post-training gains like we saw with the GLM-5 series, but you're right, the hype about being "Fable at home" was wishful thinking.

Also it's bizarre that K3 already feels old/dated.

It's unclear how big a role distillation plays, but it may be a big one. There's also a law of diminishing returns. To get a meaningful increase in model quality you seemingly need an exponentially larger model. And most people don't think K3 is actually on par with top closed models.

Google says 1 million blackwell gpus are being delivered monthly I'm really curious if it's companies just hoarding chips/memory/servers awaiting to be deployed in data centers not ready yet for months or years, or everything built is actually deployed upon delivery. Plus google and amazon have there own chips in the mix.

> And most people don't think K3 is actually on par with top closed models.

I'm honestly struggling to see the differences between K3 and GLM5.3 and the various Claude models.

Maybe Fable is better (a little), but for me at least, being able to see more of the thinking tokens is extremely valuable as it means I can steer the model quicker and more effectively.

Do you have more details on this claim? I see it a lot, and I'm trying to understand what people are doing that they see these differences.

Personally I feel we're at a point where the harness is more important than the model already. Garbage in, garbage out. Most people are perfectly fine with something as small as qwen 3.8 27b. Qwen 3.8 flash has also good reviews from home labbers. I think the diminishing returns is huge and trillion param models may not be the best use of resources.

> qwen 3.8 27b

This model is surprisingly good, I'll have to try it for some home coding projects to determine if it works for that use-case.

> what are we (in the US) even building these super massive data centers for?

Partially to make investors think it's worth giving US companies a lot of money. Also, I think distillation is a significant part of why Chinese models perform as well as they do. I think that's completely fair play (OpenAI/Anthropic/Google/Meta stole a lot of their training data). But I would expect if US model developers stopped right now Chinese development would slow down.

US companies are clearing the path, others follow in their wake.

> But I would expect if US model developers stopped right now Chinese development would slow down.

I would be willing to take this bet, given the nationalities of basically all the ML teams I've seen in the Valley/tech over the past decade or two.

There are lots of Chinese people, and some of them are super smart, so it really should be expected that they would be able to produce good models, particularly in the incredibly competitive environment of mainland China.

Can you help me understand why you think the US labs are in the lead?

> Just to churn through more backpropagation reps more quickly?

"Just" is doing an awful lot of work here. I am a simple man, sitting here, waiting for hours on end for a computer to write code for me, that I would much prefer to get in a fraction of that time.

I assume I am not very special in that way, that there are parties who have a lot more interested in getting their stuff done quicker than I do and have no trouble imagining a world that will remain intensely supply constrained for the coming years.

PS: It's actually a fantastic feature that the stuff that is used to built the machine can then (at least to some degree of efficiency) be used to run the machine.

> a world that will remain intensely supply constrained for the coming years.

That's only true as far as you're willing to pay the cost for that increased speed.

For instance, I would love to have my daily commute take 5 minutes instead of 30, but I wouldn't pay 10 times the cost for that to happen.

A 1T model with quite few active parameters though.

A 1T model with more active parameters is obviously more expensive. If it had 1T active parameters it would probably be 20 times more expensive to train.

But does that change the training requirements specifically? You're right of course for inference, unless I understood that part wrong about active parameters (which is entirely possible).

It obviously increases the time by 20 times given fixed hardware, it probably also makes the hyperparameter search more difficult and expensive, depending on how predictable the performance of your architecture is.

This was pretrained for 2 months, so a model with 1T active parameters would take 3.33 years. Maybe a 4 months pretraining run can be acceptable in a future where Euclyd or VSORA machines have been rolled out in quantity. But we don't know how far off that is.

But that (i.e. models made with 4 month pretraining runs) would probably allow an EU-made genuinely China-beating open model, so if VSORA is rolled out in quantity, let's say, tomorrow, then this could be a thing that would happen this very spring.

Most of the compute isn’t doing training anymore:

https://x.com/vincentweisser/status/2107576883298893976?s=46...

What do you mean? Is post-training/RL not training? It seems to me that inference is quite stable there and only change being more time on RL which I would still categorize as training.

You got it backwards. RL is long running inference plus training at the end. They let the Agent spin for quite a while, then they calculate the reward afterwards and modify the parameters accordingly.

... so training? Even "regular" pretraining is "inference plus training at the end" if you want to frame it that way

https://xxcancel.com/vincentweisser/status/21075768832988939...

Within the frame of that discussion, the important distinction is whether the calculation is done for a paying customer, or for model improvement. You can choose to make separate buckets for RL and training (that's interesting to understand the evolution of model creation techniques), but it won't change the end user.

For inference (Anthropic and OAI are B2C on top of B2B)

To train much larger models. It is quite possible that 10T-100T models be on the horizon

Fable (and maybe Astra) is already a 10T model.

Per a source Astra is a 4.8 trillion parameter (total) model, 5.6 sol was 3.2 trillion.

This seems to imply that training will ever be done? But yeah, I think the idea is that the appetite for thinking-on-tap will be enormous.

Even for smaller models, I think they’ve found that training an enormous, inefficient model and then distilling it internally to something much more efficient to serve is the way to go.

More compute is already being used for inference than for training.

The new data centers are mostly for inference.

One reason (not the only one) is likely strategic - deny potential competitors access to resources i.e. out spend them (competitive CapEx escalation).

Today's data centers are being built for yesterday's inference need. There's a persistent cult belief that ai hasnt found a niche or that companies havent proven utility or use cases or whatever. the demand for ai (internal to hyperscaler, and external for everyone else) simply dwarfs what is available.

Both Anthropic and OpenAI have been having major load issues though. Up until yesterday, OpenAI was serving at only 30t/s per default.

You are in violent agreement with the comment you replied to.