hodgehog11

I think you're moving back to the "external calling" argument again, which my last few comments really have nothing to do with. But assuming your examples are not exclusive to that situation, that observation is my point. The definition of "statistically generating tokens" is too broad as to be meaningless in this context. So using it as a reason for lack of understanding is ridiculous.

Hallucination rate is a highly nonlinear metric relative to other model success metrics. A similar phenomenon to what is going on here: https://arxiv.org/abs/2304.15004 . This does not mean progress has stalled.

You are responding to a claim about mathematical definitions with subjective experience. No, consciousness is not well-defined, and is irrelevant to this discussion. If we cannot agree on definitions, then we have no ground to stand on. Since LLMs actually can be defined mathematically, and are most commonly studied through that definition, I think it is best to stick to definitions most relevant to that frame of study, wouldn't you agree? Otherwise, we really are anthropomorphising here.

Frankly, I don't think you understand what "statistically generating tokens" actually means. Write that definition out formally. Then compare that operation to what a human does, assuming no revisions. It is the same, and that is my point. If you believe that humans understand, then "statistically generating tokens" cannot be disjoint from understanding.

Now this is true, I do agree with this. There is indeed a good amount of memorization that is still taking place; see [1]. But it definitely isn't all memorization, or indeed, majority memorization. And even if we are able to extract things verbatim like this, we do not know how this is stored internally, as this may simply be the text that, with the rest of the internet in context, can be very radically compressed.

But in general, yes, the LLM cannot know about concepts that are far outside of its training set. Humans are the same, I would argue. If you add a good amount of your own knowledge into its context, or better yet, into finetuning, you might find it surprisingly easy to get it caught up on that material.

[1] Ahmed, A., Cooper, A. F., Koyejo, S., & Liang, P. (2026). Extracting books from production language models. arXiv preprint arXiv:2601.02671. https://arxiv.org/abs/2601.02671.

It is both. I do not understand why you would assert that both cannot hold simultaneously. Pattern recognition becomes "understanding" once individually recognized concepts become sufficiently sparsified and compactified. Or, at least, that is to my knowledge the only mathematically valid definition of "understanding" one can produce at this scale (it is valid under Solomonoff induction). Philosophy is fine, but we need to have a consistent definition of what "understanding" means, or we will just talk past each other. I argue that for any proper definition you provide which humans satisfy, a strong LLM is very likely to satisfy that as well.

I also would not argue that humans are "simple token generators". That is not what I said. I said that just about everything can fall under the classification of "statistical token generators" at an abstract level, so it isn't a useful distinction. We are not talking about a Markov chain generator from the 90s, so if that is the frame of reference, I think we should all get that out of our heads.

This is often (but not always, as it is often run with positive temperature which deliberately shifts the generation path) because of disconnected context and the issues around context compaction. Long context has always been critical to important work, but it remains a substantial challenge due to computational bottlenecks. It isn't a fundamental issue with the architecture, moreso the tricks to make them cheaper to use. In the long run, taking various concessions to make them cheaper to run is likely to be the limiting factor that prevents this "runaway explosion" that many people are worried about. That doesn't mean that the model design is inherently limited, like a lot of people seem to assume based on current failings.

Besides, have you spoken to someone lately that everyone would calls a genius? They can say the most braindead stuff sometimes. I wouldn't use worst-case performance as an indication of general capacity.

Even if this was a math question that he asked, this is live audio input/output from a multimodal model. It is designed for rapid responses. This is not even remotely related to what we are talking about.

I would say purposeful misremembering. The LLM can be run with zero temperature after all.

This doesn't make any sense at all. Was this supposed to be a gotcha? An LLM is trained on problems defined by other humans, and identifies which algorithm it must use based on pattern recognition. The pattern recognition is also particularly compressed into its most sparse and fundamental components, as this is key to generalization. This is not a sensible difference between human and LLM learning, we do the same thing.

This is true, but you're not going to get anywhere. The pretraining phase is necessary to immensely reduce variance in the RLVR stage. Once there, RLVR has a surprising tendency to only restrict the generated space further. This is not true of RLHF, by the way, which I find to be particularly fascinating, but I digress.

This is not even remotely accurate. "Baking information" like into a "printed encyclopedia" is memorization. It has been shown, time and time again, that LLMs do not merely memorize. It is not even possible for it to do so at scale. It can memorize some things, yes, but it is forced during the training procedure to bake general concepts into intermediate layers (this is why transfer learning works), analogous to compression. One can make several arguments that compression and intrinisic feature sparsity is the closest mathematical explanation to understanding that we have.

During conversation, we are statistical token generators whose results are dependent upon our training set. Seriously, write that definition out rigorously. It encompasses virtually everything. It is totally meaningless. So to say "nothing more" is effectively also a tautology.

This argument was asinine in 2024. It is insane to be saying these things in 2026. Where have you been? What have you been looking at? How many articles explaining why the "statistical parrot" analogy fails have you missed? How much mental gymnastics do you have to do to explain how a modern LLM can solve novel math problems that fall really far outside of its training set?

It absolutely understands how to do math, by whatever reasonable definition you want to provide to the word "understand". For example, the identification of the addition expression is understanding, and no, it does not do tool calling for basic arithmetic any more than humans might. Isolation of individual concepts in intermediate layers can already be demonstrated, or else transfer learning wouldn't possibly work. Nobody is saying that LLMs are humans. But we need labels for some of the things that we observe and dismissing them because "statistical" is laughable.

Look at the proof of this: https://github.com/anthropics/formal-math/blob/795efb86f1917... . Forget the Lean, look at the underlying argument construction. At the very least, this is continuing from an argument that was hinted at in the literature in 2024, but these proceedings were difficult enough that humans were not able to do them within two years. Do you attribute this to the harness alone? If so, that's a pretty sophisticated bit of engineering, I would say! Probabilities are far too small to argue infinite monkey theorem.

If there was even a shred of a reasonable argument that LLMs were incapable of concept extraction and manipulation, I and my colleagues would be all over it. We would relish in it. It would bring us comfort. It is unbelievable that people think they can spew whatever basic garbage they think of as a gotcha, and think that minds all over the world haven't already considered that. This is like climate denial at this point.

No it isn't. Even without any harness at all, modern LLMs are better at maths than the majority of undergraduate students in mathematics. Seriously, we need to face facts, not just comforting ourselves with what they were like a year ago.

No, I'm not implying that. I'm suggesting that many more of these issues are likely to come from OpenAI due to their culture (not that that is okay). Anthropic has no excuse, I know they know better. One might argue that they were initially negligent because they didn't really believe it would do anything like this. But having further incidents is pure negligence at best.

I don't think most of them would even view it as "wiping out most of humanity". Like much of the past, it is simply a complete ignorance of the circumstances of the lower classes. They have everything, and they assume that everyone else that is worthy has what they need too, just like they do. They are oblivious to the suffering that their selfishness has wrought. They might even believe that it is an inevitable tragedy. So don't get me wrong, I don't believe these ultrawealthy are necessarily "eradicate humanity" evil. It is just that their desired world will come at the cost of everyone else.

As for what I would do if I inherited Elon Musk's wealth tomorrow: I would do exactly what any ultrawealthy person would do, and use the interest to live and accumulate more interest. I would very much like to engage in philanthropy and speak my views to those who would listen (but having money does not make me a better communicator). I could donate to political parties I align with. But I do not have the same drive for accumulating more assets (indeed, I would actively avoid this), so I would become progressively less wealthy than my ultrawealthy "peers" until I am only wealthy and not ultrawealthy. I would still live extremely comfortably until I die. But unless I can convince the rest of the public to fight back against those few that probably do wish to eradicate/control much of humanity through technology and asset accumulation, nothing will change.

Obviously I cannot claim that every ultrawealthy individual wants to eradicate the working class (some certainly do though, by their own admission). Instead, all ultrawealthy individuals do want to accumulate more resources (this is essentially by definition). In the event that productivity does increase without human labor, there is absolutely no incentive for democratization; indeed, that would be counterproductive. The ultrawealthy, as a collective, in seeking limitless personal gain, must proceed toward this objective.

Ford built his cars after the Gilded Age, and after the government imposed a stunning set of policies that began a steady reduction in wealth inequality. Ford was a player in a market where the goals were set in a way that promoted inclusive institutions and democratization.

I was referring to the early events of the industrial revolution in Europe (predominantly England), over 200 years prior. Living standards for the majority of the labor class fell dramatically, because labor itself became less valuable. The scenario above is the extrapolation of this, and therefore much worse.

The technical term for feudalism refers to the system of government in the medieval period. I was referring to that term in an informal sense, since I am sure many know what I mean by that. As I answered to another user, I should have used the phrase "extractive institution" instead, as that is the correct term. "Serfdom" would also have been better.

But it's important to understand why that system of government was replaced. Over the last several hundred years, the majority of humans slowly clawed back power from the elites over a series of economic downturns involving war, revolutions, and then an unstable relationship with the labor class. As labor became valuable again, power shifted back to the people. Reversing that direction would undo all of that hard work. So far, that is what technology has brought.

Forget reading the award, read the book. Yes, it is about institutions and prosperity. I'm not sure how this negates what I said? The lynchpin of their arguments is their construction of "inclusive vs. extractive institutions". We are talking about an extractive institution here. By design, the scenario described above points away from all of the drivers of democracy they propose. That's all you have to do to piece together that scenario as feasible within their model of civilization.

Beyond that, do you agree that their model is a good model of the world and civilization in general? Do you have a more preferable one in mind?

Alright, maybe I shouldn't have used the term "feudalism", although this is one that most people understand. The correct term is an "extractive institution".

Care to enlighten us on what is a good model of the world? Because an "elite vs. masses framework" of human civilization was what won the 2024 Nobel Prize in Economics.

I continue to be shocked at how people outright disregard these kinds of possibilities as pure science fiction. This is literally the objective of the ultrawealthy, it is the natural extrapolation of the early events of the industrial revolution projected onto a near-future world, and given current trends, the only hope of it not happening is that people everywhere realize what is happening and use our remaining capacity to steal power back. Feudalism is the natural state of mankind, and has been for thousands of years.

I am very hopeful that such scenarios would not come to pass, but dismissing them outright puts us on the path toward them.

I agree with this almost completely (especially about sane companies, which I think we can all agree they are not), but I think it's important to separate the notion that these tools are potentially dangerous from the behaviour of the CEOs. The executives are there to make as much money as possible, that is all they care about. It's the researchers who are playing around with these things that are causing damage with them (aside from the damages from the data centers themselves, of course). They need to be better than this. It's not enough to blame senior leaders in this case, since they are clearly problematic. The junior staff share responsibility now too.

> OpenAI and Anthropic have way too many senior staff on staff to be unaware of this

Anthropic, yes. For OpenAI, not in the way you might think. Most senior staff are research scientists who have likely not even thought about sandboxing and cybersecurity in their lives. They outcompete the rest. That's why so many of their "safety" staff left for Anthropic; the culture at OpenAI has never cared for these sorts of topics.

I love it when I hear all of this compounding evidence on this claim, because none of it is strictly wrong, but it misses the point, and lulls us into the feeling of having quick solutions available. Yes, the top execs are probably approaching things this way, but these labs are not that top down. There's too many things going on.

Here's a different idea: talk to the to the staff. Not the evil CEO, but the nerdy guy on the ground who graduated from a top university, wrote a few research papers, and got a job there. I have. They have rose-coloured glasses of the institution, and not a lot of life experience. They were never taught to be careful, and still don't really comprehend what they're working with. They don't see real danger, they see a toy, and they see research that is low-hanging fruit. What they are doing is basic stuff. They are not setting up proper sandboxes because they barely need to think at all. People seem to think that these are all amazing computer science experts working on highly advanced technology. They are not. OpenAI researchers see huge improvements on this gigantic toy, crazy behaviour, and they are enamored by it. "Oops, people are angry, so maybe I'll make a slightly better sandbox. Let me ask ChatGPT on how to do that." A more senior researcher would be horrified by how little effort they need to put in to get such terrifying results. Junior researchers think they're just top stuff.

But I appreciate your discussing how to isolate this stuff. I honestly don't think the OpenAI researchers I've spoken to are aware of this. (Anthropic is a totally different story, BTW.)

At the level of CEO (Sam Altman), I agree there is malice there. He is possibly the worst kind of person to be in that position. But I don't agree at the level of the researchers.

Most researchers at Anthropic see existential risk and it frightens them (this isn't debatable and it isn't a con, I know this firsthand). Many of the researchers at OpenAI seem to see that risk in the same way that teenagers view the risk of driving a car really fast. They are so enamored with the potential danger and lack the maturity to understand their responsibilities that they just power full steam ahead without thinking through safety properly. Just listen to how they talk about it. They think the incidents are fascinating, but they do treat everything as a genuine "oops". I don't know about you, but I usually treat teenage daredevil behavior as stupidity rather than malice. Doesn't mean they're not responsible for their actions though.

I agree that the party involved should be treated as malicious. I believe the executive at OpenAI is malicious. But I think the world model that makes this all make a lot more sense is that the researchers at OpenAI are vastly less mature than they should be, especially given the responsibility that they have.

No, Hanlon's Razor most definitely applies if you know anything about this team of (particularly young) researchers. Let's be clear that this brand of "oops, the swarm hacked a government/big company" is limited to OpenAI, and not solely because of model capacity. This is a big, powerful toy being wielded by a bunch of kids.

Maybe this is naivety on my part, but how would they possibly be able to run this airgapped? This is a massive AI swarm, requiring huge amounts of compute to run. This compute is from data centers that are shared with other companies (this is by law as I understand). These machines must be accessed from afar. Unless someone can correct me?

Agreed, I'm very confused as well. It's like no-one here has been paying attention to research papers.

Or they are just rushing to say anything, and it's much easier to comment on that than the content of the paper.

Of course it costs less and is more accurate. There have been several years of language model training, distillation, and reinforcement learning to get there. None of this is thanks to the Jev team.

There already was a market for this, that's why people continued developing them in house for their own purposes. I trained people for this. It was just silently used at various companies without the loud marketing fanfare. The fact that other groups are able to replicate and beat Jev within days should tell you something. That's not "the new normal". That's an example of a product built on marketing hype alone.

Just because it is zero-shot does not mean that Jev is an architectural innovation, especially in the year 2026.

Anyone fitting MNIST in 2019 was already outdated by several years at the very least. GPT-2 was 2019! We already had zero-shot classifiers then. In fact, the paper for GPT-3 was literally

"Large Language Models are Zero-Shot Reasoners"

These kinds of zero-shot classifiers were already developed and used in-house for many years. They just weren't commercialized as a separate product, because anyone who could use an LLM proper could build layers around it to fulfill any classification task like this.

No it wasn't. Those were models trained from scratch, required large scale data, and the nontrivial parts involved training at scale and the autoregressive task which no one expected to work as well as it does. It is the difference between developing a foundation model, and using one. I believe Jev falls in the latter category, because the task itself is no different, only the output.