qeternity

This is undoubtedly great. But most of the inference cost today for dominant use cases (agentic coding) are in the prefill, not the decode. This is one of the reasons that DeepSeek is so aggressively optimizing prefill and caching.

Hugged to death?

Not sure why you think everyone has moved on from this. Valid JSON is only one aspect. It almost certainly has far more uptake today than ca. Sonnet 3.7. Grammar engines can enforce a lot of other things too. Models are actually really good at emitting valid JSON, but complex schemas effectively require it.

Alternative is to fail the validation and regenerate the request, but I have no idea what this person is talking about. It is absolutely not the case that everyone has moved on from structured generation. Absurd claim.

Most people are not even referring to CUDA batching nuances.

They think that sampling is an inherent part of Transformers.

Even on this site, it is regurgitated with confidence.

It’s only good for them in the sense that it is allowing them additional time and compute to cheat a reward signal.

It is not good for them in the sense that they will short circuit the RL path that actually improves general capability.

Has nothing to do with being headless. That's just a natural outcome.

If someone wanted to use an 8x B300 as their daily driver...go ahead.

It would still be the best way to serve a given model.

*TB/s

> Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best served from a headless server so that's what I'd recommend.

What? LLMs are best served from a massive PD disaggregated cluster of B300s connected via NVLink.

If you're running LLMs on a Mac Mini, it's because you want to run local, not because it's the best setup.

You're not entitled to this stuff any more than they are.

What planet do you live on?

Trolling. GLM is heavily distilled from Gemini.

Is it not a bad thing to encourage poor people to continue doing the thing that has apparently made them poor?

It's also very not true that farmers are poor in the US (median household income about 30% greater than general population). This mindset is like a century old.

College stopped being about getting an education a long time ago.

It's just a low pass filter for the job market. Students aren't actually interested in learning anything, they have been condition for a long time to jump through the hoops.

It would be much better if we actually returned to exam and performance based evaluation of students, and stopped giving everyone As just for trying hard. But parents look at college fees as buying their kid a job opportunity, and so grade inflation has basically ruined the SNR of academic performance for all but the lowest performers.

It's the other way around: conviction rate includes plea bargains.

Convictions at trial are much lower.

You are misunderstanding the conviction rate. Only ~2% of cases actually go to trial, and at trial there is ~80% conviction rate.

There are ~4x as many cases dismissed by judges before getting to trial. And the vast majority (90%) of defendants enter into plea bargains.

Prosecutors only bring charges when they feel they have a strong case. There are many many cases which are never pursued because of this, and people also get upset about that.

Many countries do not have a plea bargain system the way the US does. And if you look at conviction rates at trial, they are smack in line with much of e.g. Western Europe.

I presume you're a Windows user then.

When the M1 was released, I couldn't believe how fast it was.

Ultimately the frontier labs are competitors of Nvidia. There is a fixed amount that the market will pay for tokens. If the frontier lab model premium collapses due open weights models, Nvidia can capture a greater share of aggregate spend.

It is still people, not money, who votes for them over and over again…

Personal hardware is only sufficient today for some tasks. As data center power efficiency, and large sparse MoE task efficiency increase, personal computing will continue to lose out.

Computer use will blow through tokens because it's doing image capture for everything.

You may have better and more reproducible results using browser controls that aren't image based, or writing tools that completely sidestep browser use.

Jevons paradox: large purpose-fit data centers increase efficiency such that you can use AI in more places, and use more tokens for those tasks.

The future is not a single chat bot session of bs=1. The future is many agents performing many tasks in parallel for a single user. Large GPU clusters will always have the edge in efficiency.

The real breakthrough is going to be thinking in latent space.

You say this like that isn’t how the vast majority of problems are solved…

Has nothing to do with Chinese.

Frontier labs have already been doing this for a while, verified in smuggled traces from OAT/Ant.

Simply a way to reduce tokens.

35A3 might be more comparable to 10 dense.

27 dense is far more capable than 35A3.

> more than half its price

Less than half its price.

More than 50% discount.

LLMs are, in theory, deterministic. Sampling is not intrinsic to LLMs.

Greedy decoding a single batch in most libraries will give you mostly deterministic outputs. Higher batch sizes can increase variance.

But all of this is down to CUDA and/or kernel implementation issues.

I don't follow this. Clearly all of the frontier labs are doing these things.

When OAI released gpt-oss it was released as an mxfp4 checkpoint.

OAI, Ant, et al are also obviously employing QAT.

I cannot be sure what the likes of Cursor have done, but I think it's incredibly unlikely that they have trained a QLoRA for Composer.

It's almost certainly full parameter post training of the original model weights.