aspenmartin

Right but the issue is users have full control over context. A security-violating action by a coding agent in one context can be completely innocuous under other contexts etc, or breaking down the task into multiple tasks that in isolation do not violate anything.

Why are we missing the point here: LLMs are of major geopolitical significance. Autonomous drones kill children, now what? Would you rather have a home grown frontier model capability to defend yourself? Or would you prefer to rely on the good graces of tech CEOs?

You are totally right, reading emails and vibe coding saas apps is all we can do. Omg why didn’t we think of that? Someone get the pentagon on the phone and cancel the drone warfare technology they’re already using and proving out in the battlefield with Palintir. Someone also call the NSA: we don’t need to hack nor do we need surveillance and spying nor do we need robust defenses against all of this. Ah!! Also the president, we can surely cede the market benefits to other countries that’s totally fine.

LLMs are a “nice to have” blows my mind. LLMs power autonomous drones that kill real people in real wars. Coding agents are the primary way people code. That and we get a GPT4->5 jump every 18 months. Are we on the same planet?

> That businesses view it as essential...is not a profitability argument.

What do you mean, as in it doesn't imply profitability (and profitability of what?) today?

> Businesses also bought dot com infrastructure, telecom fiber, crypto platforms, metaverse tools, and overbuilt SaaS.

Yes, and like I said, there is absolutely a stable state where token economics are profitable.

> The question is whether the AI application layer can charge more than its full cost and the costs are inference, infrastructure, depreciation, R&D, customer acquisition, support, compliance, security, and error remediation.

Of course they can. If they couldn't, they wouldn't, and then someone would come in and charge the correct price for it because there is a tremendous amount of demand that will spur supply. I'm not sure what the mystery is here.

> The numbers so far do not inspire confidence. OpenAI reportedly did $4.3B in revenue in the first half of 2025 while burning $2.5B, and Microsoft said OpenAI related losses reduced its own quarterly net income by $3.1B. An MIT 2025 enterprise AI study found $30 to 40B spent on GenAI with 95% of organizations seeing zero return.

I do not find this particular argument inspires any confidence either -- everyone is trying to capture market share. How on earth, on a blog with ycombinator literally in the URL, do people not understand that lack of profitability now says absolutely nothing about profitability in the end state, after this growth phase?

You seem to imply by your MIT study citation that there is "evidence" that AI is not even profitable. Reading between the lines here, your theory is: the entire market is wrong, AI is a net negative ROI (and in your mind, for how long? forever?), and is inherently unprofitable. That is a crazy thing to think in light of all of the datapoints we have.

> One of the core technical reason is that hallucination destroy enterprise economics. If SAP hallucinated 2% of invoices, or Oracle returned fake rows 2% of the time, nobody would call that early stage friction. They would call it unusable for core operations.

Yet we see absolutely incredible adoption? In your view we'd see a spike and then all of those dumb CEOs will say "oh oops we were dumb" and adoption will go down. Hallucination does not destroy enterprise economics. Hallucination rates, by the way, have continued to decline after every model generation. Believe it or not there are incredibly valuable, smart, sustainable ways to incorporate AI. Even if it's just coding agent licenses, that alone is a powerful enough revenue driver.

> In legal AI, even specialized tools have been measured hallucinating 30% of the time. The problem is that as AI gets better it is confidently, plausibly wrong. That forces humans to verify it.

You seem to be conflating two things: one is that there is an inevitable eventuality where everyone will finally shut up about AI being unprofitable because we exit this growth phase and reach a more steady state market economics. The other thing is that we are not yet there. I'm not arguing about us not yet being there, though I do think the evidence even there is a bit cherry picked.

> So the cost does not disappear. It moves from doing the work to checking the work. AI coding has the same issue. If an autopilot got you there faster but one flight in ten became unstable unless the pilot constantly supervised it, that is not productivity.

That's just not going to be true for long. The reason you see coding agent adoption and other step changes in adoption + capability is that error ("hallucination" and other error modes) get to a low enough point where they are acceptable for a given application. This will eventually eat most applications. Yes we have to check things today, but this will be less and less important once we reach a point where humans are even less reliable than AI at _checking_ output from either themselves or other AI systems.

> For the bull case to work, the usage must explode, the quality must improve, prices must fall, reliability must rise, legal risk must shrink, and margins must expand and all this at once.

What??! this is all happening _right now_ right in front of you. Usage _is_ exploding in ways we didn't even dream about 8 months ago. Quality _is_ improving in very predictable ways (scaling laws anyone? benchmark trends?). Reliability _is_ rising _and thats why you see the adoption level you see for coding agents_.

You can do what frontier labs do today which is to properly license things that are copyrighted and use open source web crawls for things that don’t have copyright issues. You can then also commission new datasets (volume needed goes down when quality is high).

The European regulations are the thing that will kneecap anything meaningful coming out of Europe. Mind blowing to me that this is worth the tradeoff since Europe will be beholden to other frontier labs be it China or the US, so regulations accomplishing very little if anything on impacting actual AI development and losing vast amounts of leverage in the process.

You can grumble about the way things are but Europe being so far behind a technological race with important geopolitical ramifications means that you guys are cutting off your nose to spite your face. Have fun with zero of European regulations impacting frontier development and then eventually having to depend on it.

You know what I'm surprised to find out this is far more feasible than I assumed; DiLoCo + INTELLECT models demonstrate how feasible decentralized training is already, that is very surprising to me that you can get that far with so much less communication bandwidth. Not only that, but that distributed training is _more_ feasible as you scale since compute needed scales as the square of parameter count but communication scales linearly so the overhead penalty goes down.

I think the most important problem is that you have to marshall enough compute to be meaningful, and that is going to be more and more difficult as frontier compute requirements grow.

> The US government basically has to nationalize AI and capture an outsize portion of the revenue from it in order to fix the economy, as the combination of debt burden and interest rate pressure from de-dollarization/global realignment is going to push us into a death spiral, and even if AI is a smash hit, the ~19% federal capture of corporate revenue isn't nearly enough to pull us out of it.

Any actual numbers to back this up? I don't see how nationalizing a very cutting edge technology outside of wartime is going to go super well. The leverage that these companies have is the same leverage that TSMC has: you can't just take over and expect things to rocket at the pace its going

It's the most naive opinion that keeps getting shoveled around. You have a product that is viewed as essential by businesses, with revenue growing by 10x a year and geopolitical ramifications that have continued to rear their heads and your opinion is "this is all an unprofitable shill". It is extraordinary to me that people really believe this. Whether or not labs run at a loss today is absolutely irrelevant. There is of course steady state economics that make sense, and its currently not well known what the profitability picture is right now, so to say "Currently AI has generated no profit" is also just speculation and not a very insightful one at that.

It's more than the raw hardware, it's the interconnect and communication between the hardware at scale. These models are trained on hundreds of thousands of GPUs today. You _will_ start to see cross-datacenter training runs but this needs to efficiently decide when and how to communicate across datacenter, which bears a very high cost compared to intra-datacenter communication.

Yes but not violations of the laws of physics. You need extremely fast communications, memory bandwidth, etc; you cannot get that with distributed training. You're up against the speed of light and the interconnect that powers the internet. You will always have horrifically slow latency compared to if you pack the servers together in the same place with specialized networking.

This is really a feel good argument and I agree with what he’s saying in principle but it offers zero in terms of a practical strategy or stable state where this is feasible. If you want to jump on the bandwagon then let’s put our pants on and offer a concrete suggestion that is practical and coherent. Otherwise what are we doing. Does anyone have suggestions to that effect?

What’s the world in which frontier model performance is open source? What does that look like? What’s a sensible business model that makes this sustainable? What’s a sensible regulatory framework that doesn’t hamstring AI progress?

Everyone is so enamored with these Chinese lab models like deepseek and qwen and GLM but they exist in a world where the top performance is still claimed by closed source models. These are not developed out of any benevolent commitment to the principles laid out in this article. A world in which OSS is the frontier and its development is controlled and funded by government subsidies of an autocratic government is not reassuring. You can inspect weights but good luck getting the cat back in the bag in terms of capabilities, safeguards, value system, bias, nerfing if it smells American business use cases.

Deepseek was such a darling but guess what, it’s now raising money — 300M at 10 billion valuation. OSS development isn’t sustainable as a business model and in a world where it costs a few hundred million to develop a frontier model, you need a strong business model, or you need strong state subsidies and incentives which introduce a billion new problems.

the most sensible economic picture of OSS models already exist. Commoditize your complement, passion projects for a hedge fund. These are unsustainable and exist at the pleasure of the business or the founder.

This is really wonderful -- One thing that may be really cool if you have the data is to add a time-axis ability (unless I missed it) for a given location. This is such a delightful application of AI!

Absolutely. All of these "but you could have done that easily" from frontend developers or backend developers or systems engineers -- like yea, if I have the time or interest in those things, sure. But I don't. I care about an end product way way more. Blows my mind that there are legions of people building things that they don't think are important enough to get to the finish line quickly and efficiently.

Those are some incredible sprinkles.

Oh ok well then you’re definitely right about that, they can tell and sometimes it really matters (I can’t remember if it was SWEBench or not but there was a major benchmark where the models were just inspecting git histories that were leaked into the dataset). The more insidious one is alignment but idk alignment research that well to know if this is a big deal or not.

Yes but so what right? This is a problem for both alignment evals and actual cheating (e.g. someone forgot to delete .git history and the model was able to back out the original PR, or they can decrypt something by finding a key, etc), but both of these are beyond the scope of what I'm talking about. The impact on these evals that are affected is small, and so what if you know you're being evaled when I ask you to give a new proof for a conjecture? I just care whether or not you can do it...

They don't have control over measurement. Consider also it's easy to figure this out and it creates a scandal. Like I said, consider Llama 4 which a lot of people pointed out used a custom model in LMArena to inflate their scores; its never clear what the true underlying story for this, but regardless that model release spurred billions of dollars of spending on new talent and a complete gutting of that org.

These companies have to care about good measurement frameworks because the quality of their models depends on it. Any PR department can polish a turd, but an army of smart researchers far outside the control of these companies are going to figure it out if they are gaming metrics.

Its not a limit scenario is my point: these models are evaluated constantly, new benchmarks both public and proprietary are in constant development, benchmarks are not always static either, they can often times be living benchmarks that update over time.

You are making a technical point, which I am pointing out that while for _some_ benchmarks this is _technically_ possible, it's not true for plenty of benchmarks that all agree with the others.

> which of course would mean that the benchmark was created entirely "by hand" or using some other provider that is unconnected to the provider you are benchmarking

yes this is incredibly common. I'm not talking about hypothetical scenarios.

> To put it another way: a closed-weight model is, by definition, impossible to independently benchmark.

Even if you believe this, you're doing some mental gymnastics if you think this is really the most likely explanation for what we're seeing. It's absolutely possible to benchmark proprietary models when you don't have access to the weights or control over the API, even if they are adversarially trying to combat this, which they aren't. Doing what you're describing would be easy to detect: you'd see extremely high benchmark scores for established benchmarks and then poor scores for new benchmarks as they come out. It would be relatively easy to figure this out and not subtle.

Yes I've seen this before, and while the critiques are fair and high quality (and unfortunately not unique to METR) we're missing the forest for the trees here.

First of all, if you take the articles critiques and work out the implications on the METR graph, all you're doing is shifting the curve up or down, it doesn't change the fact that progress is scaling exponentially. While it is technically possible the universe could be throwing a massive pathological curveball to change the conclusion from METR data (which is we've been seeing exponential growth over the last 6 years), I think that seems very far from likely. The fact that we see the same behavior from a variety of sources over a wide variety of tasks and domains is a pretty clear indication that METR while certainly far from perfect is actually painting a consistent picture at least in terms of the rate of progress.

You can look at ECI for a summary benchmark statistic, which does NOT use METR's benchmark, and you see a similar trend. Same with SWE-bench where the task distribution is far more in domain for real world problems. It is a bummer that this METR data can't be better funded. It would probably take $1M or so to really beef it up properly which any of these labs probably have in their couch cushions.

I appreciate the data here but I don't think the read is quite right;

Saying we have linear capability for super-linear cost compares an unbounded variable (dollars) to bounded instruments (because benchmarks saturate). On unbounded measures, growth is exponential; you can see METR time horizons double every ~4-7 months (https://metr.org/blog/2026-1-29-time-horizon-1-1/). And capability being proportional to log(compute) is what the scaling law predicts.

Epoch puts training cost growth at ~2.4x/year as your link shows. Meanwhile cost for fixed capability falls ~10-40x/year (https://epoch.ai/data-insights/llm-inference-price-trends), and lab revenue is growing ~10x/year! Anthropic went from $1B to $9B to $30B+ run rate in ~15 months, OpenAI ~$25B.

On [3]: the "destroying value" conclusion flips sign on an assumed 15% baseline rework rate. The report's most direct metric is +16% merged PRs per dev. The RCT evidence is genuinely mixed (METR: -19%, with n = 20 and Claude 3.x; Cui et al: +26%) but its just super hard to do this well, I think Faros stuff was pretty cool, I haven't seen this before so thank you for the reference.

Some people sure, but some people believe the earth is flat. What’s without a doubt is the impact AI has already had, something maybe 2 years ago was dismissed as religious fervor maybe. Statistics and robust measurements plus stable scaling laws make this pretty far from religious I would say.

Ooof a Yann LeCun quote from '24, a risky time period to be mining his quotes from lol. But I think broadly he's right, but the argument is: we don't see any evidence for the curve flattening, and we should plan for it _not_ to flatten any time soon. "It will eventually slow down" is true but its meaningless if you dont put any sort of time period on this.

> Tech people are following a religious belief system whose utopian promise is the all-powerful computer that will end all suffering.

Uh, I don't really think that's anywhere close to an accurate characterization of most people here. Everyone, including Dario and any researcher at any frontier lab, knows the situation is quite scary and unprecedented. There are problems that will be solved and diseases that will be cured, but will we be living in an Orwellian universe? Will a rogue drone swarm find you cowing in your basement and murder you? I mean the technology for this is already mostly here, it's a matter of the willpower and budget to roll out something really evil.

The comment's question is about capabilities and why the discussion about capabilities often times is far removed from todays capabilities.

I think at this point the "instilling" part is not nearly as challenging and thorny as "what values should we instill"; that part is hard to imagine going away as it feels pretty fundamental to humanity that wars have been fought over.

It's nice that people are genuinely curious about this.

- All of your observations are absolutely dead on

- Yet, we have very very very robust scaling laws that as Dario points out we've had and validated for over a decade. This extends to downstream measures like METR time horizon and compsosite benchmarks like the epoch capability index.

- If you look at where you're at now, which is again dead on, you're looking at a point on a curve that is quite easy to extrapolate, but less easy to tell when exactly on the curve a certain capability or use case undergoes a step change from error rates dropping below a threshold that is hard to anticipate in advance.

So while Dario / other frontier CEOs are understandably unpalatable, they are absolutely spot on with a call out that all of this is bound to happen and happen quickly, and that's without solving several core problems that haven't been solved yet (e.g. continual learning). In 2023, coding agents were just laughable. Yet they followed the same predictable training curves. Anyone looking at the data can see the obvious, and anyone reading newspaper headlines or hacker news comments would get a very different impression.

> The hype is coming to an end

I'm sure you have evidence for this

Why does this have anything to do with what I’m saying, of course the models are updated. I’m saying a new benchmark isn’t public and the model wouldn’t know they are being evaluated on a new benchmark.

Not to mention: thinking that the api behind the scenes is literally swapping to overfit models to maintain some sort of illusion that they perform well on these benchmarks is just beyond ridiculous.

> Sure you can, just do it silently and don't tell the people hitting your API that the model is different now. Unless it's open weight, we're just taking your word for it. Even better, do a VW and try to detect which benchmark is running, then change to a hyper specialized model that is trained on it.

This is...just incredibly conspiratorial and a bit silly. You can make a benchmark right now and run it on the models. They'll have a benchmaxxed model on your...previously non-existent benchmark? I mean: if models really were overfit to benchmarks, which zero lab is doing because its idiotic, against their incentive structure, and easy to detect, then why would we see a slow ascension of performance on say humanity's last exam for one benchmark example? You could trivially get those numbers to close to 100% if you wanted to.