hodgehog11

Indeed! Almost none of these most math folks are likely to read. The only thing to read is the statement, which is only a handful of lines in both cases. The point of Lean is that if that statement compiles and is validated by hand to be equivalent to the natural language statement, then it is true. That was what I was trying to say here.

That's fair, but I would argue that reading the specification is very easy by comparison. A quick one hour tutorial is usually enough judging from my students' experiences.

Why do you believe that learning Lean is a better use of time when the AI is clearly better at writing and interpreting Lean than it is writing quality papers? At this point, Lean is for autoformalization, no one is really supposed to read it.

Do you care to share then? Because anyone I know with any sense right now is aware that "planning for the next forty or so years" is genuinely impossible at this point, especially in mathematics, and the right approach is to remain on your toes, ride the waves, and diversify as much as possible. Heck, even planning for the next five years seems absurd. This is a time of immense uncertainty, and to think otherwise is foolish.

Yes. Most probably do not understand the notation involved in, and the statement of, the Lindeberg-Levy Central Limit Theorem. But every scientist uses this theorem in one way or another. These ideas have a way of trickling down because to people who work thanklessly to do so.

"Plausible" text was preferred over rational text when we trained LLMs using RLHF. It's rapidly shifting the other way now with RLVR, which enforces correctness by default.

No, this is different, and this is coming from someone who has been studying deep learning for the last decade. We are talking about the difference between RLHF and RLVR strategies. The former benefits clarity and explanation, while the latter concerns only correctness. AI was moving in a particularly damaging direction by pushing on the first path, so it was natural to move to the second. But the second will come at the cost of clarity of explanation. It will likely get better at its explanations, but not fast enough to render its most advanced accomplishments readily understandable to the user. The chess example is a pretty good one (that is an RLVR approach).

The first mistake that you are making is assuming that OpenAI is a singular conglomerate with the same set of values across its members, and that its researchers bear the same set of behaviors and opinions as its scummy executives. It isn't. The OpenAI researchers are probably a group of their mates, and they have every belief that they are being just as honest as any of their other colleagues. These antagonistic relationships do not really exist between research institutions; that is an invention of outsiders and marketing departments. Everyone is from the same group.

I don't see the report as absolving the industry of wrongdoing. In fact, I believe it is particularly damning, and many of my colleagues feel the same. No one is on the side of OpenAI here, not even its own members.

As for submitting the report back to OpenAI for comment and working with them, given that there is currently no government mandate or requirement for any AI company to disclose wrongdoing, this seems like an unfortunate necessity. Independent bodies and governing agencies have to be created. At the moment, it is the industry regulating itself, which means no regulation at all.

So no, you do not need to believe that the document is truth. But it is all that we will likely get, because nobody with any power will do a damn thing. That is just as frustrating to the researchers as it is to you. The solution is not to defund the institutes doing needed research. It is to put laws or incentives or something in place to punish inappropriate behavior and lack of disclosure for the companies.

Many of the most prominent people in the AI industry all know each other or have deep social ties. Why? Because they all came from the same places. Neural network research was niche circa 2012 and early adopters post AlexNet were few in number, especially compared to today. This by itself does not infer a conflict of interest. If you believe so, then you might be surprised to find that all research is like this in every field. The most prominent players were part of the initial niche. Sub-optimal, yes, but this is the reality.

Now, I do not know METR, and have not interacted with its staff. $120m is pretty normal for these types of groups as I understand. METR clearly has even tighter ties to the AI companies than any of the groups I do know, so I agree that calling them independent is a bit of a joke. Even without the explicit labels from their website, there are shockingly few degrees of separation in this entire field. The conferences are big, workshops are designed to be highly social, and the stakes feel high.

Regarding the timeframe and limited material, that's pretty standard now. You do what you can.

Now, none of this necessarily invalidates what they did. Yes, there is big money in AI, and that money hires people, but there quite a lot of geeky researchers in these groups that genuinely only care about the science. That level of neurodivergence might be difficult for most people to comprehend, especially if you're focusing on the politics, but these are the types of people attracted to this line of work.

Put yourself in the shoes of these researchers. You are really interested in AI and how it works, and you convince yourself that this is potentially moral to work on because of a possible threat to humanity. You apply to work at one of these groups. You find similarly-minded people and work together on trying to figure things out. Maybe your boss is a sleazebag in bed with the industry, or maybe they're not. You do not care. You work on this, you publish notes and papers, you comment on LessWrong or X, and you leave the company if they don't let you do this stuff, because there are dozens of others.

Most of the public don't know these guys or this work, because they only focus on the crappy and irresponsible companies leading the AI development charge, and alignment research is so unbelievably hard and slow. I can go into the why if you're interested, but the reason is that it is extraordinarily difficult to make progress if your objective is only vaguely defined. This is the same problem with the Yang-Mills mass gap, for example, although I would personally bet that Yang-Mills has a much better chance of being solved than alignment. Just a different tier of difficulty altogether.

I also disagree that p(doom) work (I don't think this describes more than 1% of the work out there, closest I can think of is Tegmark's group, which put it pretty high from recollection) advances the pace of AI. I don't believe it is doing anything. Nobody externally believes it anyway (why would they?). These companies are run by narcissists who believe that they should be the ones to reach "superintelligence".

I think most supervisors try to discourage students from writing in this way because there is a delicate art to it that you are unlikely to be able to meet at that career stage. When I read some of the texts from Martin Hairer and Cedric Villani in my field (or even Riemann's famous paper on the zeta function), you can see that they are motivational and informative but still concise at the same time. Full of personality too, especially Villani. That's a hard balance and it requires exceptional understanding of the topic and the reader.

At the same time, we really should be encouraging it more. I have found that in newer machine learning theory papers (strictly theory, not empirical work), there is something closer to a good balance that is expected even of students.

Including a motivating example is key to this, and should be considered mandatory.

> The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry.

This is unbelievably ignorant speech. I have not received a dime of any of this funding, but I do know many excellent researchers that have, and they do fantastic work. There is an unbelievable gap between theory and practice regarding the capacity of deep learning, and while great strides have been made to develop the surrounding theory, there is a long way to go. Many believe that without a concrete understanding of how neural networks properly learn concepts, we have little hope of molding them to be reliably useful. It costs money to hire researchers and develop fundamental theory.

Just because you don't understand any of that work, does not mean that it is pointless. This is fundamental research that is 20 years behind schedule.

Maybe just nostalgia talking, but the height of Newgrounds really seemed like a golden age for light gaming. The shift to mobile games has just resulted in a large number of gambling simulators, which I'm sure has been a key reason for the popularity of Kalshi, Polymarket, etc.

This is a fantasy I remember hearing in grad school. Get a little more senior and the whole perception shatters. You can put whatever rubbish you want out there. But if you are getting attention, or want attention, you have to play by the rules. Those rules are whatever the community dictates. Otherwise, why would Nick Polson be at risk of being discredited? It's no different from any other field, math is not special. Correctness is a necessity. It is not sufficient.

This was my point though. Not sure why you are disagreeing? The original argument was that there was no criteria for publication other than replication, or that is how I read it. I was refuting that.

Perelman was already working on an interesting problem, and wrote in ways that were less formal, but still academically sound, so of course he did not need to publish it. What about the people working on problems that are not interesting? You don't think they encounter significant difficulties getting their work out there?

Absolutely not. Otherwise, if I prove something original, and it is correct and replicable, this would mean I can publish in Annals of Mathematics. Now if we narrow that to publishing somewhere, then sure, but that's irrelevant. This is about values for prominent results (e.g. Annals) for mathematicians.

Care to explain? If a paper is not interesting, it would not get past reviewers. Unless "interesting" is not part of the criteria for entry to the journal? That sounds interesting if true, but that certainly has never been my experience.

I feel like I am going crazy, since this is taught in most academic writing courses. Being replicable or persuasive or correct is not enough. It needs to be perceived as valuable, or it will not get through. I learned that the hard way, and found others that taught this message only later.

No you can't, have you published in top journals before? You can only publish what the community finds valuable, period.

Nah, it's the whole benevolent dictator mentality that these socially maladjusted individuals believe. If you believe that you are the only good guys, you are likely the bad guys.

It is not deciding control, it is determining agreed values in policy form. At least, that is how I see it.

Actually, for what you are mentioning, it is getting worse. There was a sweet spot somewhere around the release of GPT-o3, and ever since, the LLMs have been getting more accurate at solving problems, but worse at explaining how, and to hone in on what is interesting. This isn't surprising, as RL strategies shifted from RLHF to RLVR, so priorities during learning changed. I don't expect AI labs to reverse course on this. We can expect AI proofs to become increasingly incomprehensible over time.

He could not freely disclose and publish his results, that was the whole problem. He only got traction when one result was automatically validated because Hardy had already seen it. His findings more broadly were only considered fine once they were understood and verified. This declaration is basically the same thing. Having Lean code doesn't substantially change that.

It's almost like a lot of non-mathematicians are commenting and have no idea about how the field really works. Or at least, how it works at the top level.

Math is not merely a tool. The question isn't one of doing the math, it's about publication and release of work. Humans have to follow certain rules, so why should we allow AI labs to behave differently?

The issue is generally not accuracy in my experience (humans are even lazier about this). The issue is that LLMs do not understand what is interesting and what is procedural.

I've spoken to researchers at both OpenAI and Anthropic. They are not "line workers". The researchers at Anthropic are more senior, that's all. None of the folks I met, from either institution, would likely know how to properly airgap their runs. I don't, I'm a maths geek, not an engineer. Honestly I wouldn't even have thought to look it up. Maybe that's a hiring problem.

Both companies have not been equally reckless, but they have both been reckless to varying degrees. I don't have any personal stakes in either company, and would prefer both to go bankrupt if I'm being honest, as I despise their behind-closed-doors attitude.

Other than that, do you think AISI is reckless too?

We cannot engage in an intellectual discussion if we do not agree on definitions. So far, your definitions of understanding seem to be whatever vibe you are going for in the statement and I would urge you to think about what a sensible mathematical definition of understanding is so that it can sensibly be assessed on neural networks. Otherwise, this isn't science, it's a debate about personal experience.

> Understanding is a state of mind

This is meaningless, it is a circular definition at best.

> It exists entirely within the individual and nowhere else

Then why are we talking about it? What is the point if it is something that can only be defined per individual?

Regarding your language example; this is a case of missing vocabulary (excluding grammar of course, but I feel like that is second-order), which is not the same as conceptual understanding. We are often able to translate because we have shared concepts. Those concepts are what we really care to assess with LLMs.

> requires interpretation by a person to "know" an LLM "understands."

We are still not getting anywhere because you have not prescribed criteria to determine whether it understands. If it is a "know it when I see it" situation, that clearly isn't working. For example, if you say that you need to dig into its internals and figure out whether it is breaking things down appropriately, that doesn't work because you probably don't have the expertise to do that. The experts that do are telling you that it very likely understands because it pulls apart most concepts in the way we would expect.

I do object to the use of the word "simple". "Statistical" is so broad to be almost meaningless; it merely means that a prediction is being made in the presence of data which possibly contains some degree of uncertainty. "Simple" encompasses that which can be understood readily by a non-expert.

Quantum mechanics is statistical (this is literally the Born rule), but evolutions are not operating as stochastic processes in the sense of Kolmogorov. That is very different, and not relevant to our discussion.

Yes, of course it was aggressive. It is frustrating to experience so many armchair experts on a forum usually populated with intelligent people, regurgitating debunked arguments from years ago, which get in the way of educating people about what is really going on. See the recent Hoog video for how frustrating this is. I believe this is how the climate scientists felt.

And yes, according to our best definitions, the Robin bird does understand the worm it's pecking at.

I think you're moving back to the "external calling" argument again, which my last few comments really have nothing to do with. But assuming your examples are not exclusive to that situation, that observation is my point. The definition of "statistically generating tokens" is too broad as to be meaningless in this context. So using it as a reason for lack of understanding is ridiculous.

Hallucination rate is a highly nonlinear metric relative to other model success metrics. A similar phenomenon to what is going on here: https://arxiv.org/abs/2304.15004 . This does not mean progress has stalled.

You are responding to a claim about mathematical definitions with subjective experience. No, consciousness is not well-defined, and is irrelevant to this discussion. If we cannot agree on definitions, then we have no ground to stand on. Since LLMs actually can be defined mathematically, and are most commonly studied through that definition, I think it is best to stick to definitions most relevant to that frame of study, wouldn't you agree? Otherwise, we really are anthropomorphising here.

Frankly, I don't think you understand what "statistically generating tokens" actually means. Write that definition out formally. Then compare that operation to what a human does, assuming no revisions. It is the same, and that is my point. If you believe that humans understand, then "statistically generating tokens" cannot be disjoint from understanding.