Hacker News Reader

Comment by 9cb14c1ec0 | Hacker News Reader

Just a reminder for everyone that we are only several years and 3 or 4 iterations into hardware being optimized for LLMs. We should all expect orders of magnitude improvement in speed and/or cost over the next 5 years. Then we can have fun conversations about "unlimited" "intelligence" and about what the price wars and profit margins of consumer AI products are when your average ChatGPT user costs the company $0.10 per month.

> CS-4 delivers more than 1,000 tokens per second on models exceeding 10 trillion parameters

Wow!

View on HN

And, the software side isn't finished being optimized, either. We've seen with Qwen 3.8 27B and DeepSeek V4 Flash 0731 and GLM 5.3 that quite small models can pack a punch. Intelligence density will improve, efficiency of kernels will improve, efficiency of KV caching and MTP will improve, algorithms for splitting workloads across compute units will improve.

It'll all be as cheap as DeepSeek was before the price hike. And, it'll become more and more realistic to run near-frontier intelligence on personal devices.

Hence why taalas was one of the best strategic acquisitions of the year.

I'm honestly baffled they were not acquired by somebody else (sorry AMD).

Taalas will be one of the great disaster investments of the early AI era. It'll be a near total write-down.

The absolute worst market time to etch a model to a chip is right now (very rapid iteration). There is no scenario where they can keep up. The Taalas approach will be viewed as comically foolish within just a few years.

Cerebras will win in terms of approach.

It's 1998: hey, I can drastically speed up your web service, let's etch it right to silicon.

It's 2026: let's etch nginx into silicon and get 10,000,000 rps at a cost of 0.1 US/day.

Yes, please!

Most people probably don't care about nginx performance. It shouldn't be your bottleneck unless you serve massive amounts of static data.

Ok, how about postgres?

In the case of needs to process natural language, instead, massive efficiency (esp. time) can be a game changer. It's like "you have two years to complete the project" vs "you have two hours to complete the project": if you can squeeze that "two years worth" into a negligible delay, it's a game changer.

maybe AMD wants the IP to deploy it once ai model development slows down in a few years. Or, their large cloud customers do want to burn through silicon, basically paying rent to AMD for models etched on silicon.

I just want to but hardware so I can run a model at home that is fast. I don't see myself installing a server that burns almost two hundred kilowatts but maybe a card which runs a 27B Qwen...

At 250w when it's working (I understand), and it works for tiny amounts of time per query...

Corrige: a Taalas HC1 PCIe card consumes actually between ~200W and ~250W (a 10 blades server consumes ~2500W).

> The absolute worst market time to etch a model to a chip is right now

Slightly disagree. It really depends on the price-point at which they can do that etching. ~1k usd / ~30B model in a hdd-sized case that fits on your desk? I'd buy one right now, even knowing that I'm "stuck" with whatever model of the day is.

Time to market also matters a ton. If they can start shipping chips <1 month after the weights drop that's much more compelling than if it's a 6+ month development pipeline.

In the case of Taalas, the pipeline was said to be 2 months:

> From the moment a previously unseen model is received, it can be realized in hardware in only two months ( https://taalas.com/the-path-to-ubiquitous-ai/ )

I would still pay ~500 for a chip that runs 10kt/s of a ~100b model on my machine even if the half life is 6months. I m sure my employer would too.

Qwwen3.5 122b was released 6 months ago and is still best in class overall 100-140 B param model.

10000%

> I would still pay ~500 for a chip that runs 10kt/s of a ~100b model on my machine even if the half life is 6months.

....

Considering the 8B model uses 53 billion transistors, that's 6.625 transistors per parameter.

https://taalas.com/products/

Assuming they can get it down to 3 (somehow), that's still 300 transistors, or 5.565 RX 9070s.

https://www.techpowerup.com/gpu-specs/radeon-rx-9070.c4250

You're looking at

1) waiting for another 3-5 generations of transistor improvements before it can fit into a single conventional chip, or

2) another generation before getting a monster of a chip (1000+ mm^2), and prices for flawless etching scale quadraticly (likely $1000+ for manufacturing costs alone).

Could happen, but it's a long shot for a market that could be satiated by specialized accelerators.

In Taalas HC2 a chip embeds 20b parameters, and the declared idea is linking the chips. A card with two of them chips and you can already have a dense Qwen at staggering speeds.

Chiplet-style layouts could cut the etching quality requirements per chip down, but it still can't avoid the base cost for manufacturing silicon.

> it still can't avoid the base cost for manufacturing silicon

What costs are you talking about and why would them be a problem?

If it is the price: «Kharya says it costs 100x as much to train a model then to get a customize HC chip in reasonable volumes from Taalas» ( https://www.nextplatform.com/compute/2026/02/19/taalas-etche... ).

That is thinking about an LLM (logical) producer and server. For mass production, the costs go down. And in the case of a ~100b model as the poster mentioned, they would be just single PCI cards with 5 or 6 HC2 chips: doable and practical.

I see more potential problems in the positioning of the SRAM - but not a real problem given that excellent team.

To get a proper idea of the costs the architecture of the HC2 will have to be clearer.

500 what? You’re missing the unit

Etched model into a chip? A… mobile chip eventually? Seems prescient.

The 500x efficiency gain makes their approach a no brainer. Just make a new chip every 6 months, you still win.

Yes, power efficiency may be the largest benefit of these chips actually - especially if the projections are true that the US and other countries simply aren't able to ramp up power generation to meet forecasted datacenter demand.

There's a bit of a ticking time bomb there, something that a Taalas-like architecture can clearly resolve.

> It's 1998: hey, I can drastically speed up your web service, let's etch it right to silicon.

I distinctly remember 32-bit/33 MHz PCI accelerator cards for SSL being a real thing (for use on OpenBSD or FreeBSD), in an era when something like a single core 700 MHz Pentium 3 1U system was a relatively powerful individual bare metal httpd box.

http://www.aster.si/partnerji/compaq/atalla/axl200.html

The CPU load of doing a lot of SSL purely in software was a problem in terms of scaling things up, so this was one attempt at a (very short lived) solution. Note that this predated TLS1.0.

> Then we can have fun conversations about "unlimited" "intelligence" and about what the price wars and profit margins of consumer AI products are when your average ChatGPT user costs the company $0.10 per month.

We can have that discussion now: sounds like that would kill OpenAI and Anthropic

A design that bakes the architecture into silicon would be 10x faster, and imagine a version that does all the multiplication ops using single log-amp addition versus dozens of transistors to cut down the amount of silicon used by 50x. The ceiling for AI optimized hardware is extremely high.

Stack on top of that the fact that diffusion based models like the ones made by Inception Labs are far faster and more efficient than autoregressive LLMs and have an even higher ceiling of optimization (single step path prediction via model distillation versus 50 step denoise is currently an active area for image diffusion)

The human brain is soon neither going to be more powerful nor energy efficient than the stuff we use to run AI.

This is part of why I think the data center build-out is a bubble. We've barely scratched the surface when it comes to hardware optimization. We'll see exponential improvements in energy efficiency and speed over the next decade. Exponential, not linear.

GPUs really aren't that great for AI. They just happen to be the best chips we have in mass production right now for this work load, and it takes time to field new designs. Basically every chip engineer on the planet is working on this right now.

On the plus side, lots of cheap servers to swoop up :)

But power hungry.

In that 5+ year timeline, the compute per watt could change by three orders of magnitude.

GPUs are to LLMs what CPUs are to gaming — not a good fit.

A cursory estimate courtesy of ChatGPT suggests that there is a grand total of one order of magnitude or less of power efficiency improvement available compared to current Blackwell if the entire system’s power consumption outside the ALUs went all the way to zero.

If you want three orders of magnitude improvement, you probably need to find two of those orders of magnitude somewhere else: process improvements, different ALU design, model architecture changes, etc.

Oh, you could go analog rather than digital. Ever look at ALU design? Nothing efficient about it!

Look at their power supply, it’s not something you can run in a home lab. Unfortunately most of that will likely go to the bin eventually :(

No but dedicated server prices will likely drop or at least you'll get more for your money

Whether it's a bubble or not depends on how much the demand for compute and the type of workload keeps growing, though.

If AI tends to be something used mainly in ideation and development, which is how a lot of people use it today, then once consumer hardware gets good enough you could see a bunch of the current data centre workloads move onto consumer devices.

But if AI starts being used more in repeatable, operational workloads I think it makes sense to have significant cloud infrastructure for it. TBH I haven't seen much of this, and I've been skeptical about people using agents for much of anything when it can be done with just software. But we are starting to see more of this kind of workload, like the taggable Claude in your slack etc that people seem to really love.

I wonder: in world where inference is cheap, how many engineering agents that use simulation as their feedback we will use?

In the scenario, engineering everything becomes so easy - so why not optimize everything? every component, every product, every system?

And maybe llm's could invent. So even more to simulate. And simulation is inherently compute-heavy.

So unless there are some other bottlenecks, we'll use a lot of simulation servers.

True, I have to agree with you. The AI giants might be investing a huge amount of money in generation 1 technology. There might be a much better way to do it just around the corner. They might know this and thus the hurry to IPO.

A rough analogy would be if the first generation of ISP's spent billions on dial-up exchanges, when fibre could be invented next year.

By the time these gigawatt datacenters are done being built the hardware will be so far behind state of the art they may be mostly useless.

Exactly right, and nVidia is protecting their moat through business practices rather than genuine product innovation.

By the way, this is the same argument that Michael Burry used to short Nvidia.

He claims that GPU depreciation/obsoletion is much faster than hyperscalers are assuming because new chips will be much better. He's being proved wrong right now because H200 rental prices have been claiming for the last 8 month despite B200 having 10-20x better inference efficiency.[0]

The logic is fundamentally flawed in my opinion. Let's use future Nvidia chips being much better optimized for LLMs for example.

New Nvidia chips 10x better than H200 --> data centers buy a lot --> Nvidia profits a lot.

New Nvidia chips 10x better than H200 --> data centers don't buy --> no faster than expected obsoletion.

In other words, the very act of buying many new Nvidia GPUs would be the event that causes faster than expected obsoletion. Yet, if you don't buy those new Nvidia GPUs, then there is no faster than expected obsoletion.

We also live in a world where there is competition. If Amazon doesn't buy but Microsoft does, suddenly Microsoft can offer better $/token prices.

[0]https://inferencex.semianalysis.com/inference

1. The same isn’t necessarily true of the rest of the hardware stack which may be reused between accelerator generations.

2. You’re missing the “New Nvidia chips 10x B200, compute requirement grows less than 10*software improvements YoY -> buy less Nvidia.” Valuations are based on forward projections (>1T annual for NVDA) which can be revised down leading to a drop in valuation.

> If Amazon doesn't buy but Microsoft does

The big 3 all have their own proprietary accelerators. Meta is buying TPUs as well for now.

I would bet Nvidia’s major customers in 2 years are neoclouds and it seems that Jensen is making the same bet.

1. So this makes Burry’s argument even less convincing since those auxiliary hardware can last longer.

2. Jevons Paradox. More efficiency should lead to bigger models, faster inference, and more total tokens.

3. By all accounts, Trainium and Maia and Meta’s internal chip are struggling to keep up with Nvidia. That’s why they order as many Nvidia chips as possible. They’re not giving up but it isn’t as easy as buying stock Arm cores and taking them to TSMC.

Neoclouds may very well be Nvidia’s biggest customers and this probably what Nvidia wants.

1. Not really, current valuations are priced for persistent 80%+ margins based on spot. If auxiliary hardware lasts longer (I.e. next gen GPU reusing the same shell) then that reduces supply pressure and spot prices.

2. Jevon’s paradox is about total consumption, not margins. Valuations are about margins (and their projections). Many coal mine owners went bust despite increased total coal consumption.

3. Source? Gemini for example is 70% on TPU. I have yet to see data on Maia-300 beyond Microsoft PR. Remember it doesn’t have to be better it has to be more cost efficient. The overwhelming majority of inference spend does not care if token output is 20% slower if it is 50% cheaper.

> Neoclouds may very well be Nvidia’s biggest customers and this probably what Nvidia wants.

What Nvidia needs. Whether neoclouds can stay competitive vs hyperscalers paying Nvidia tax is far from clear, particularly when inference margins compress.

1. The whole Burry argument is that AI hardware becomes obsolete faster. If aux hardware can be reused, that works against the argument.

2. Total consumption drives more demand for the already supply constrained hardware. Can AI hardware market go bust? Sure it can. But being early is the same as being wrong in the investment market. When do you predict the bust to be?

3. Google, Amazon, Microsoft, Meta are all buying as many Nvidia GPUs as they possibly can. The biggest tell on how Nvidia is doing is that their share in inference has increased despite the increase in competition: https://archive.md/CKP0N. So while competition is getting bigger and bigger because the overall pie is getting exponentially bigger, Nvidia's growth is still higher than average.

Some other sources:

https://www.businessinsider.com/amazon-nvidia-aws-ai-chip-do...

https://www.businessinsider.com/startups-amazon-ai-chips-les...

> 1. The whole Burry argument is that AI hardware becomes obsolete faster. If aux hardware can be reused, that works against the argument.

Burry’s main argument is depreciation is being understated and the capex vintages will not be paid off before they are essentially useless. This can happen whether or not aux is reused.

> Total consumption drives more demand for the already supply constrained hardware.

Demand is the wrong metric.

Only number that matters is whether AI attributable revenue will be sufficient to pay back enough of each successive capex vintage (e.g. 750B this year, 1T next year, 1.2T in 2028) so that hyperscalers and neoclouds can either self-fund or continue to issue debt as bond markets are already straining and tax-payer backed sovereign debt is providing a high baseline. Otherwise they downgrade capex projections and the bubble pops.

Expensive compute needs expensive inference to justify 30-40B/year/GW of compute. There are many reasons why frontier API pricing which is what the industry is based on may not persist. It is also almost certainly the case that 2026 is the worst year of supply and demand mismatch to allow for 80%+ margins. HBF next year has the potential to single handedly pop the DRAM spot bubble.

> Can AI hardware market go bust? Sure it can.

This is the bear thesis. It is not that AI will crash or be useless.

> But being early is the same as being wrong in the investment market. When do you predict the bust to be?

Q4 27-Q2 28 is when the bill becomes due at the latest. There are sufficient financial levers left to buy time without returns until then.

> Google, Amazon, Microsoft, Meta are all buying as many Nvidia GPUs as they possibly can.

All of these companies have rock solid revenue streams and can easily swallow 500B of capex devaluation over time. Their buying of Nvidia today is not necessarily the indicator you are implying as there are strong competitive reasons to make the game more expensive for everyone else.

  Burry’s main argument is depreciation is being understated and the capex vintages will not be paid off before they are essentially useless. This can happen whether or not aux is reused.

And why does he think depreciation is understated? It is because he thinks newer Nvidia GPUs will make older ones obsolete faster. Hence, my entire post.

The rest of your argument centers around whether AI growth will meet the cap ex expenses. I don't see anything new in it.

  HBF next year has the potential to single handedly pop the DRAM spot bubble.

I'll believe it when I see it. Jevons paradox will apply here again in my opinion. HBF does not replace HBM.

> We'll see exponential improvements in energy efficiency and speed over the next decade. Exponential, not linear.

Why? Make your case.

What LLM-specific hardware improvements should one expect? Seems to me that LLM inference is simple architecturally (matmul et al) so most scaling in hardware should come from general improvements (memory BW, packaging, interconnect, power).

What you describe is basically Cerebras case, at the bottom it's just a really big die (about x28 an NVIDIA GB200) with a lot of work to reduce memory latency and improve throughput. What it's actually amazing is how can they make a chip so big and still have a decent yield to be commercially viable.

Congratulations! You have just realized that the AI data center build out is a total scam, built on both the insurmountable trillions of debt, and the assumption that only GPUs are all we need to continue scaling.

There exist other AI accelerators (TPUs, ASICs) that perfectly exceed the throughput that LLMs need to scale as well. But the true solution is more software optimizations. There's a tiny handful of them but more needs to be discovered so that we can reduce building hundreds of more data centers as the alternatives mature.

As better software becomes more useful for the alternative AI hardware for developers with LLMs running efficiently you then would have more choices of hardware to run your LLMs on rather than just only GPUs.

TPUs and ASICs run in data centers too. Your argument only holds true if there's some satisfied limit to demand for inference. If not, data centers will continue to spring up to host more and more agents. Even if agents were running on hardware and software as efficient as the human brain, its conceivable we want trillions of them running at any given time which would require data center scale.

Everything has some satisfied limit to demand, often depending on the price. If you assume there will never be any satisfied limit to demand for inference at any price you can justify any investment.

Yeah, but there's certainly a part of the curve where price drops by X OOMs and demand increases by much more than X OOMs. (Presumably some of that is substitution and some of that is new use cases.)

Looking back nearly 80 years, what has been the limit to transistor demand so far?

Unlimited.

What has been the limit to electricity demand globally?

Unlimited.

We can't get enough and never will. Costs have to become pretty severe to turn back the demand as well.

So far, at least by Openrouter's weekly numbers, there doesn't seem to be a satisfied limit.

https://openrouter.ai/rankings#top-models

And their market share sits at around 16-20%.

At 75.3 trillion tokens for the week ending 10 Aug 2026, that means that up to 450 trillion tokens were plausibly demanded by the whole market for that week.

My take: At max saturation, each person on earth could have their demands satiated by an average of 16 agents running concurrently. Sometimes more, often times less, but the average would likely be at 16.

At 200 tokens/second for each agent, that would mean 15.48288 quintillion tokens per week.

We're currently at about 0.00290643601% of the calculated demand ceiling.

Even if the demand limit per person is just 1 agent at 50 tokens/second, the current demand's still 0.186011905% of the theoretical ceiling.

I wonder what this looks like in 5 years... Will there be a massive push to repurpose these giant boxes into housing? Will they get turned back into the farm land from where they came? When a data center goes bust, what happens to the parts left behind?

I'd think the infrastructure would tend towards factories, smelters, and so on. Industrial things that have reasonably high power demands, can use the building, and don't care about the lack of windows.

They're typically not built where you want housing, and the buildings are distinctly the wrong shape.

If you can't use the power infrastructure profitably my next thought would be warehousing.

But also... we've seen a pretty continually increasing demand for compute. Even if AI busts a bit (or becomes a bit more efficient) I bet most data centres stay data centres, just less profitable ones.

Huh, why I'm not surprised that HN is full of opinions confidently stated without any numbers or resources to back up?

> built on both the insurmountable trillions of debt, and the assumption that only GPUs are all we need to continue scaling.

Insurmountable according to whom? And who assume that only GPUs are all we need to continue scaling? Google, Amazon, Microsoft, Meta and OpenAI, all have or plan custom non-GPU AI chips. Do they plan to use them not for scaling?

Is this article not exactly about ASICs built for LLM inference and training acceleration?

Reply on news.ycombinator.com