aspenmartin

This is just a misconception of how LLMs work and also what reasoning is.

“There cannot be any reasoning embedded in the model” a strong statement, what do you mean by reasoning because by any reasonable definition I’m aware of, they clearly are able to exhibit reasoning.

The fact that the pre training objective is next token loss has nothing to do with capabilities or their ability to reason. To be highly successful at next token prediction you NEED to reason. I’m quite confused here.

“It sounds like you believe” is a question, inviting your clarification. I will continue doing that because it’s perfectly reasonable. Also “machines have no driving force to act in the world” is also a mysterious statement but because you reacted so badly to anyone questioning you I will just leave it at that

“They don’t understand” is a strong statement, maybe true but depends on what you mean by understand. What is your definition of this? I can’t think of a meaningful definition of “understand” that doesn’t apply to LLMs

Yea it’s rejection sampling, so you have an agent, you take a verifiable problem (people use lots of different verification signals but say unit tests etc) and have the agent attempt it K times. You accept the trajectories (all context, tool use etc, the entire log) that are positively verified and use these as training examples.

The trick is to find the examples that are just in between too difficult and too easy for the existing agent, these have the strongest training signals

You’re describing the problem of continual learning. As I said their “consciousness” for lack of a better term and knowledge does already change from ongoing experience in context which is another of saying for only a short window, today. They are ephemeral, sort of, but that’s a temporary limitation.

That’s strange to me, what would you define as learning?

Same with humans? Doctors, scientists...if a tool has any error rate above zero its not reliable?

You are implying definitions that don't seem to be mainstream; thinking is internally manipulating information to reason, infer, plan, solve problems, and form judgments or beliefs. Also -- "Without a prompt an LLM does nothing. It has no thoughts between prompts about you or your problems." it sounds like you paint this like it's something fundamental? It isn't. Nothing is stopping you from streaming information to an LLM and letting it process this information, this is precisely what people are trying to build.

They do learn in context, and very sample efficiently. Continual learning is active area of research and we sort of already have something resembling it with persistent context. So yes they do learn.

Right but hallucination rates have been consistently decreasing every model iteration. It's about error rates. As also a fellow scientist, I also will mess something up. Humans have an error rate. Once that error rate is low enough, it doesn't matter that it's > 0, it matters that it's low enough to be trustworthy and useful. Coding agents of 2024-25 had error rates too large; you couldn't meaningfully vibe code anything and needed a ton of oversight. It's still true but FAR less so, and this is after like a year of iteration.

Data curation is important and expensive and frontier labs can afford to do it right. Natural data isn't the limitation, we are already literally out of tokens. It doesn't matter how much you poison things it's not going to stop the progress train.

It's not really a problem. We're out of natural tokens anyway. The future is synthetic verifiable traces (already the way we train coding agents).

If you get to E you have generated a Bayes-optimal model of the conditional distribution (as in, next token conditional on context). This is something I thought too, but even if you're a fraction of a nat above the floor, you could have enormous headroom in performance left because there are still rare tokens amongst the irreducible noise that require so much capability to predict. It's not to suggest there truly is no cap on capability, but just that this constant isn't really saying what that is.

Right but that is context dependent; it drops with context length, depends on tokenizer, etc. It doesn't end up being super relevant, despite the fact that if you look at the loss for real models it's relatively large in absolute terms. But that doesn't really matter -- all of the interesting stuff happens once you start getting closer and closer to it. You've gotten past all of the easy tokens that dominate the entropy and now you get to the really challenging ones that we care about (like e.g. very difficult reasoning about a next step).

I really wish more people skeptical of AI capabilities would read about scaling laws -- Lilian is always so marvelous at giving a deep overview of the technical side but the whole point of this is: there are scaling laws, and they hold and continue to hold. This is such a huge basis for the predictions about AI capabilities for the past like 5 years.

> Every year since 2023 the models are too dangerous to release and in 12 months all white collar jobs will be obsolete. This might not have been a deliberate lie but it's clearly been untrue and they've said it again and again.

How is a prediction a lie? Did they tell you "this will definitely happen in X time"? Their speculation is not only valuable (they are the closest to the technology) but also necessary (they need to buy long term compute contracts so these predictions are literally what they have to bet their real money and company success on).

They have said again and again that this will make an incredible amount of tasks obsolete, and they are of course right about this. The models _are_ dangerous to release, every time we hit the frontier. This has become _increasingly true_.

> Predictions with wrong timing are frankly worthless.

Who cares?

> I predict at some point in the future the S&P 500 will be at 10,000. Of course I'm guaranteed to be right. But have I really predicted anything useful?

You aren't cherry picking and strawmanning here? Should we have a tour of all of the things that have indeed been predicted well and already come to fruition? Was 2025 "the year of agents"? It very much was, wasn't it? Additionally, unlike the S&P, performance trajectory, for almost a decade, is incredibly stable and predictable. It's hard to know, a priori for a given task or category of tasks, what specific error rate will trigger a phase transition but it's absolutely obvious and clear that this will happen quickly. It has indeed happened quickly. Does 2026 coding look anything remotely like 2024?

> If Dario was really worried about protecting the sheep, he wouldn't cry wolf every five minutes because everyone knows that's the worst possible thing to do.

No you're right he would make well reasoned arguments for the types of problems we need to address urgently. Hmm...that feels pretty ethical.

> If Dario was really worried about protecting the sheep, he wouldn't cry wolf every five minutes because everyone knows that's the worst possible thing to do.

I don't feel either of them are trustworthy, they are CEOs acting in their companies best interest. But people suggesting Mythos delay was some sort of PR ploy is some of the most extreme mental gymnastics I've seen. I listen to the actual words spoken by these people and consider the hard data that is in abundance at this point. I listen to the large body of research on alignment and safety and measurement that anyone can read for themselves or use AI agents to digest for them.

Be specific, what are you talking about. Industry has been continuously warning about many of the complex problems that are going to happen as a clear consequence of the technology. I don’t know of any problem they have talked about that hasn’t either already come to fruition in one sense or another or that just hasn’t yet arrived. Dario has been predicting the end of coding for a long time now and look where we already are.

So yea no it’s more like it’s important for industry leaders and those closest to model development to proactively identify the issues that they don’t have complete control over or that we don’t have a regulatory framework for.

Super puzzling to see these comments and of course with zero specifics just “they’re all liars and grifters”

Well the good thing is you’ve done the homework to definitively demonstrate that this is remotely true. These confident claims of this all being some sort of unprofitable Ponzi scheme, not understanding the concept of a growth phase which a multitude of highly successful tech companies have already demonstrated work while simultaneously commenting on a site with YCombinator in the url are just getting amusing now.

Of course this is a profitable technology, and it doesn’t matter if any of the labs are profitable today or not. Running at a loss is a perfectly rational strategy.

What was the dog and pony show?

> They were actively trying to make the doom happen while trying to cash out maximally on doom trolling.

These words make no sense. Anthropic delayed mythos/fable rollout. A mythos model without safeguards would have been a pretty bad idea, and they sacrificed a ton of revenue and risked being scooped by any of the other labs in the meantime. Frontier models are only frontier temporarily until the next lab releases their model. Of course they are a company and need to act in their own best interest.

It is also clearly serious the problems we need to think about as we march quickly towards even more capable systems. Why on earth is it a problem to point this out?

> If they were actually concerned over social impact, they would try to minimize it. They could have sell their product as a tool to be used to make economy boom, they tried to sell it on promiss to make it shrink for most people.

What a really weird take; they employ some of the best safety and alignment teams in the industry and this is an active area of research that they are campaigning for more attention on. You complain about them “doom trolling” and then complain they don’t do anything about…the doom? No sense at all.

It is perfectly consistent to (1) sound an alarm and (2) March full steam ahead as quickly as they can. If they don’t do (1) that’s unethical. If they don’t do (2) someone else will. I would rather someone like Dario align these models than the CCC. Plus it would be nice not to have a war over Taiwan which is inevitable if China gains enough of the upper hand in this AI race.

Be specific, what do you consider their lies to be? Also, this is pretty straightforward. You have a decade of extremely stable and predictable performance trajectory. It’s easy to see the writing on the wall. You can feel whatever which way about their motivations and ethics but if you read say Dario’s raw words they are pretty reasonable. We have to have a good regulatory framework and do what we can to prepare ourselves while also not ceding a critical strategic advantage. The west coast is always cult like, that’s not new. And it ignores the very real substance to the discussion.

Well I would say you just chose ANOTHER sensible path with tradeoffs just like frontier API/subscriptions. Yes you get 100% control, cheaper inference, not having to stare at status.claude.com for like 2 hours per week, complete privacy (assuming local hosting or self hosting on servers).

But, you can’t get Fable level performance. OSS has reliably trailed the frontier by like 4-7 months for years now

My favorite part is how people compare AI to crypto as if they are remotely in the same universe of importance and impact

And in response to this we have a very robust UBI system for horses

Absolutely, you are right even though many may consider this too snarky. But herein lies the fork in the road.

- consider that right now autonomous drones operate in Ukraine using the latest and greatest frontier models

- consider that fascist or authoritarian governments will now have a kind of access to their citizenry’s deeply personal lives that they’ve only dreamed about before. “Show me everyone who has insulted me” is now feasible to do at scale, today. A system that can monitor, in real time, a global sigint network on every single person equivalent or better than having a dedicated person or team of people monitoring them. Every home has microphones, every square inch of street has satellite imagery, security cameras, etc.

Now, given today where we are: what do we do? Do we regulate ourselves into a situation where people feel better about things like data center construction but that impact our ability to compete, ceding these technologies to other countries that would gain an incredible amount of leverage and power over us? Do we continue unabated, damaging communities and industries in a fervor to be at the top that later down the road we realize is not worth the tradeoff?

If you mean the technology diffuses into all products and processes instead of some standalone production that maybe makes sense to me but the models themselves feel like a product

To me I don’t really see this; electricity, the internet, etc: aggregate employment numbers don’t change work just gets shifted around. But your workers at the very least will be way more productive and that’s the sell.

Yes but what evidence is there that they aren’t? To me, scaling laws and empirical benchmark trends along with high adoption (all of this together, not individually) strongly support this, and also coding agents have become load bearing so it’s sort of already happening I would say.

Does it need to?

Frontier model companies aren’t banking on the general public “wanting” AI. They are on a path to a product that won’t need to be wanted because the entire economy will need it.