why_only_15

doing a u32 compare instead of an f32 compare is not rust-specific or indeed CPU-specific.

This trick is very useful on Nvidia GPUs for calculating mins and maxes in some cases, e.g. atomic mins (better u32 support than f32) or warp-wide mins with `redux.sync` (only supports u32, not f32).

Probably more like richest 10%, which applies to most people in the US

Random unrelated point: in a 100km radius circle between Atlanta and Augusta there are ~2,000,000 people (calculated using https://www.tomforth.co.uk/circlepopulations/ )

I'd be pretty curious to get patio11's opinion on why #5 happened.

my claim is not that it would take two days for such a pipeline to run but that it would take two days to make an NLP pipeline whereas an LLM pipeline would be faster to make.

Assuming the 10M records is ~2000M input tokens + 200M output tokens, this would cost $300 to classify using llama-3.3-70b[1]. If using llama lets you do this in say one day instead of two days for a traditional NLP pipeline, it's worthwhile.

[1]: https://openrouter.ai/meta-llama/llama-3.3-70b-instruct

how is this a plateau since gpt-4? this is significantly better

True, though it's still enough nutrients to feed three people's entire caloric needs.

What is this typically called? Are you referring to raw milk? I think this is a different thing.

What he means is that 10M cows, all the cows in the US, produce dairy products that are 10% of calories in US diets.

Fantastic article. I didn't realize dairy cows lactated ~60lbs/day or ~3.5% of their body weight. Totally insane. Chickens appear to be this way too -- from some quick research Rhode Island Reds are ~3kg, lay ~300 eggs/year and each egg is ~62g.

The graph at the end ("US milk yield continues to grow, but falls short of its genetic potential") is interesting. If I saw that graph, I would interpret it as dairy farmers overfitting on the genetic yield potential measure, not as something that needs explanations like climate change.

those cows probably still exist right? you can still get milk from them. Do they taste better?

Used to really love his stuff, but this is all pretty well-trod ground and he doesn't have much new to say.

I'm not too up to date but as I recall there are a lot of weirdnesses because of how big their chip is (e.g. thermal expansion being a problem). I believe they have a single giant line in the middle of the chip for this reason. maybe this makes HBM etc. hard? certainly their chip would be more appealing if they cut down the # of cores by 10x, added matrix units and added HBM but looks like they're not going to go this way.

they have about 125GB/s of off-chip bandwidth

My understanding is that they mask off or otherwise disable a whole row+column of cores when one dies

the cost is not the memory technology per se but primarily the wires. SRAM is fast because it's directly inside the chip and so the connections with the logic that does the work is cheap because it's close.

Why would converting a specific LLM to an ASIC help you? LLMs are like 99% matrix multiplications by work and we already have things that amount to ASICs for matrix multiplications (e.g. TPU) that aren't cheaper than e.g. H100

Not enormous without significant changes to the ML. There are two pieces to this: improving efficiency and improving flops.

Improving flops is the most obvious way to improve speed, but I think we're pretty close to physical limits for a given process node and datatype precision. It's hard to give proof positive of this, but there are a few lines of evidence. One is that the fundamental operation of LLMs, matrix multiplications, are really simple (unlike e.g. CPU work) and so all the e.g. control flow logic is pretty minimized. We're largely spending electricity on doing the matrix multiplications themselves, and the matrix multiplications are in fact electricity-bound[1]. There are gains to be made by changing precision, but this is difficult and we're close to tapped out on it in my opinion (already very low precisions (fp8 can't represent 17), new research showing limitations).

Efficiency in LLM training is measured with a very punishing standard, "Model Flops Utilization" (MFU), where we divide the theoretical number of flops the hardware could provide with the theoretical number of flops necessary to implement the mathematical operation. We're able to get 30% without thinking (just FSDP) and 50-60% are not implausible/unheard of. The inefficiency is largely because 1) the hardware can't provide the number of flops it says on the tin for various reasons and 2) we have to synchronize terabytes of data across tens of thousands of machines. The theoretical limit here is 2x but in practice there's not a ton to eke out here.

There will be gains but they will be mostly focused on reducing NVIDIA's margin (TPU), on improving process node, on reducing datatype (B100), or on enlarging the size of a chip to reduce costly cross-chip communication (B100). There's not room for a 10x (again at constant precision and process node).

[1]: https://www.thonking.ai/p/strangely-matrix-multiplications

The number for that is I believe 1 terabit or 125GB/s -- 21 petabytes is the speed from the SRAM (~registers) to the cores (~ALU) for the whole chip. It's not especially impressive for SRAM speeds. The impressive thing is that they have an enormous amount of SRAM

Why does portability matter here? I guess if you want to run on macOS too?

I'm confused about optimizing 7 watts as important -- rough numbers, 7 watts is 61 kWh/y. If you assume US-average prices of $0.16/kWh that's about $10/year.

edit: looks like for the netherlands (where he lives) this is more significant -- $0.50/kWh is the average price, so ~$32/year

People typically quote google search as ~300k QPS

To some degree this is true, but he’s also FOIA’d documents that describe what’s officially sanctioned.

In the post he mentions why he thinks this is unlikely and is not a thing the US has done previously.

They're not saying

Specifically the clause is that you cannot use their consumer cards (e.g. RTX 4090) in datacenters.

If you are making a prediction about the world, why not gamble on it? It shows to others that you have an actual stake as opposed to blathering.

If you are making predictions about the world, you should be willing to put money on those predictions so that you are grounded to making predictions that you actually believe in as opposed to just blathering.