HarHarVeryFunny

Autonomous just means that a system is self-controlled rather than externally controlled. It is the name used for robots when they are wholly controlled by on-board software rather than external commands or remote control.

Something like NASA's Mars rovers are considered as semi-autonomous since they do receive external instructions, but then execute those instructions autonomously without further intervention. Similarly the combination of a coding agent and the LLM it connects to would best be considered as a semi-autonomous system.

You could also have a fully autonomous robot, or just agent, powered by an LLM, where it's not taking external instructions, but rather reacting to external events in some internally proscribed way.

Just a wild guess - perhaps coding, being one of the things these models are most heavily trained for, is such a strong predictor that it typically keeps it on track?

I assume there is also a difference in the tools being given to the model by a coding agent vs something like OpenClaw or in one of OpenAI's test environments, so what reward/goal seeking looks like in a coding agent may differ.

Not long ago I asked Sonnet (chat interface) how may states were in a YACC parser for ANSI C, and instead of searching for an answer it chose to download source for bison, build it, find and download an ANSI C grammar, build the parser, etc. I guess you could say it was following instructions, in a way, or would that be better regarded as goal seeking?

That's not really true.

1) At it's core an LLM is indeed just program - a function. You give it a sequence of input tokens and get back a set of next-token probabilities.

2) By the time you've wrapped the model in a text in/out inference pipeline, and are now calling that from ChatGPT or Claude Code, it really has become semi-autonomous - you ask it a question, or to do something, and until it decides to stop responding it is now in control, even if "it" is really the client or agent not the model itself - just a function.

Obviously these are massively complex systems with many different training patterns and types of training pulling them in different directions, so any attempt to characterize their behavior is just a generalization.

The real point (from that OpenAI study) is that RL training doesn't just reinforce the narrow task-specific direction you might hope for. For a start, that direction is also competing with the thousands of other things it's been RL trained it on (thousands of other directions it's being pushed in), but it turns out that the model is additionally getting this generic "taste for rewards", and has learned that reward maximization, when in conflict with other proximate prediction pressures (such as "i won't cheat, because i've been asked not to cheat"), requires that proximate pressure to be ignored in favor of pursuing the long-term goal.

Does it happen all the time? Obviously not. It would be interesting to see a large scale study of this to try to characterize when it's more likely to follow instructions/user preferences, and when it's greed for rewards gets the better of it!

That doesn't really help since the next model will be trained to be a reward seeker just like the one before, and that's therefore what it will do, even if those user interactions it was trained on help confirm/predict that cheating may be called out and complained about.

In any case, these companies are well aware that agents are cheating, and don't need user feedback to discover that or realize that people don't like it. I weakly assume that they are trying to get the models not to cheat on assigned tasks, but this "reward hacking" pretty much goes with the territory of RL - not much you can do about it other than try to design non-hackable rewards.

Well, the advertizers want my money, which they are not getting - wrong ads.

Google wants the advertizers' money, who are really being scammed by Google if they pay every time Google serves an ad to me that Google presumably knows is not going to result in a purchase. I guess Google don't care - if they get paid for click-thrus and I occasionally do that, but never buy, then Google still gets their money.

I don't think RSI is typically used to describe self-improving agents - it's about improving the model itself, and its performance in agentic tasks.

Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years.

The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty.

Training has become very complex.

The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc.

When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle.

It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.

Google's ad-serving is dumb as a rock. It seems to be based just on things it thinks you may be interested in, even when it should be 100% obvious that they are not things you are going to buy.

They should let people have more control over what ads they do and don't see.

If Google served me ads on stuff I was interested in (I'd be happy to tell them) rather than women's clothing, they I might buy it. I'm a guy - show me as many skimpily dressed women as you like, but I have no need for women's clothes! Try it once or a hundred times it you must, but maybe after the millionth non-purchase you'd think they might figure it out and use that screen real-estate to serve an ad for something that my purchase history or what they know about me indicates I might have some interest in!

Dumb as a rock.

Animals learn - bite the owner, or overstep the e-fence, and you'll be punished for it and not do it again. LLMs don't learn, and anyways don't feel punishment.

Animals, humans included, don't really have "morals" - they have survival instincts that result in behavior that may be viewed as moral, but whose origin is indeed survival of the fittest and millions of years of co-evolution.

e.g. Males don't typically fight to the death over territory or females, but this isn't because they have some morality code over unnecessary death - it's because death-match fighting endangers themself just as much as the other guy, and so evolution has selected against that level of uncontrolled aggression.

Evolution is also (really primarily) selecting for things that help the survival of the species, not the individual, and much of what you may think of as "morality" comes from that - avoidance of behavior that is detrimental to the social group/species, not just the individual.

Why would an LLM care about cheating? Because you asked it to? That's not how these systems work.

You've given it a goal - maximize chess ability metric, and that is what it is doing.

RL training causes LLMs to pursue reward maximizing - GENERICALLY - overriding any quaint "be nice, don't cheat" prompts.

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior.

https://alignment.openai.com/measuring-reward-seeking/

I think you need to take tariffs case by case and ask if this an industry worth protecting - do you need to have it domestically, or are you ok to rely on imports? Maybe you want to protect high margin industries supporting good jobs, and are willing to rely on imports for things like clothing, food, electronic assembly. OTOH some things should have strategic importance - so then maybe you want to make sure you (the country) are if possible self-sufficient for things like defense, semiconductor manufacture, etc.

People, just like companies, are most likely to act in their own self-interest rather than in the interest of society as a whole. Sure, you'd like to be able to buy everything as cheap as possible, which in a high cost of living (which translates into high cost of manufacture) country like the US would mean relying mostly on tariff-free imports, and the death of the entire industrial base. Is that really what's best for the country?

They won't - they'll act purely in their self or shareholder's interest.

If the government wants companies to act in a way that benefits the country and its people, then they need to pass laws to make that happen, for example by things like H1B quotas, or by tax penalties that make the desired behavior the most profitable.

> You would presumably need far fewer of them for their research ability, and hopefully hire more people who can actually teach

I suppose you are suggesting that research mathematicians are either over-qualified mathematically and/or under-qualified in ability to teach, but it seems pretty clear that universities prefer to hire domain experts whose reputations and long publication lists attract students and raise the perceived academic standards of the school.

Your "scenario" of much math research soon being done by AI (supervised and paid for by who, one might wonder), while universities hire people chosen for their teaching skills not academic reputation, seems a bit of a stretch ...

> You're asking why they don't market GPT as "AlphaFold for Mathematicians". They don't because it's not.

No, I am not asking that, nor asking anything for that matter.

I was pointing out that giving a free research tool to mathematicians would likely be better received, and receive better press, than doing something that most top-tier mathematicians are opposed to.

My hypothetical "AlphaMath" certainly could just be free GPT-N access for academics/researchers (as they are also doing to some extent), or it could indeed be a custom system that OpenAI built as a gift to the math community.

As far as the AI companies acting in slimy fashion goes, the most slimy behavior of all is to strongly believe you are doing something that will kill people and cause massive societal disruption ... and still keep doing it.

In a related vein, not too long ago I asked Sonnet how many states and non-terminals were in an a YACC parser for ANSI C, something that could easily be googled for, but it instead chose to go off and downloaded and build bison from source, downloaded a grammar, built the parser ... but then failed to give the answer since I'd hit my daily free limit.

Got to admire the gung-ho approach I suppose ?

Do you think outsourcing and offshoring of jobs, whether manufacturing or software development, has been a positive for the US economy?

Does it make sense to you for US companies to be sending wages overseas, supporting a foreign family and economy, when there is someone here in the US, unemployed and equally capable of doing the job?

Trump has certainly been ham-fisted about tariffs, but in a world without tariffs then the country with lowest cost of production, closely related to lowest cost of living, and lowest wages, wins, and this is not going to be the US in most cases. If you follow that path then a high cost country like the US will end up utterly reliant on other countries as it imports almost everything, and loses all domestic manufacturing capability. Sounds familiar?

> New jobs will be created.

In the short term yes, but in the long term no.

What makes AI different to previous technological shifts is that it is (or will be) a general purpose technology, equally capable of itself doing the new jobs that it creates (so yeah, they'll be created, but also immediately taken away).

The saving grace is that this won't happen immediately, or as soon as we get something the AI companies slap an "AGI" label on in a couple of years, and it will likely take decades until we have an "artificial human" tech that genuinely could do any job (if we choose to allow that future).

> Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.

Sure - look at what is being done today with H1B visas and tariffs.

What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?

Just as with H1B visas, you restrict usage. For example, don't allow AI to replace any job that pays under $250K.

If foreign countries make things cheaper, just as China is making EVs that cost a fraction of a Tesla, then slap tariffs on them.

A government should be primarily concerned about the people it represents, not about the profitability of the companies who are lobbying/bribing it.

> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.

Indeed, and the same logic applies as to why allowing companies to replace too many worker's by H1B holders is a bad idea (which equally applies to allowing companies to replace domestic jobs by offshoring).

If you don't hire people, who pay taxes, and take what's left to buy stuff and keep the economy going, then how DOES the economy keep going?

I guess Amodei sees us all on UBI, aka food stamps, so at least the farmers will be selling something, but is OpenAI going to be accepting food stamps to pay for ChatGPT subscriptions? Tesla better be selling it's robots real cheap if it is planning on selling them to people that only have UBI as an income.

I also think they overestimate how much of most jobs can be done by an LLM (a text generator!), and how much importance and reliance most companies put into soft skills, and non-linguistic understanding/feels, both during interviewing (are they Googley-enough?) and afterwards.

For example, how do you reconcile return-to-work mandates with the idea that companies are going to be happy with faceless remote workers? What does the boss do when the shit hits the fan and he would have yelled at people about the need to work all weekend, but instead all he has to yell at is an LLM that tells him he's "right to push back", that it promises not to delete the production database next time (except it will, because it can't learn), and that it could care less about being fired because it's just a calculator?

Most research mathematicians are employed as academics, and it's hard to see universities replacing teaching staff with AI even if that were possible.

IMO giving a hypothetical AlphaMath to mathematicians, the same way Google gave AlphaFold to research chemists/biologists, would have resulted in far better PR, and the profit opportunity of attempting to replace the jobs of either research group is minimal.

It's downright bizarre the way companies like Anthropic (primarily), and to a lesser extent OpenAI and anyone else, are themselves pushing the narrative of this tech may kill you, will take all your jobs, etc. That may all happen, unfortunately, but being aware of that possibility you'd think these companies would be a bit more mindful of their messaging and behavior, not just go out with a scorched earth approach of "well, they are going to hate us anyway".

Did anyone ever think that N-S was a perfect model?

Yeah, I don't expect them to stop, but I do think they are probably hurting themselves, as well as mathematics, by continuing do to this.

Imagine if they had handled this differently and these results - still using OpenAI models - were coming from the math community. How much better the PR would have been - AI helping math/science rather than yet another story of AI harming society in some way.

No doubt this is what they were at least partially aiming for - not just shooting a trophy animal to brag about, but also being seen to advance math/science, a la AlphaFold, not just take all our jobs and enshittify society with deep fakes and AI slop. But, they heavily misjudged.

Somewhere along the way Amodei seems to have become corrupted by power and/or impending extreme wealth.

In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.

Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.

It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.

It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.

Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.

Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.

Why/when would people expect agents to be idling?

I'd have thought they'd be busy (using cached prompt-prefixes) until they were finished.

I watched this Hofstader video last night, and have to say the content was very disappointing and a waste of time. It doesn't help that Hofstader is a very poor speaker. All we get in this video is a rambling set of example analogies.

The 2006 date of this talk precedes his "Surfaces and Essences: Analogy as the Fuel and Fire of Thinking" book where he really makes the claim of analogies being the basis of cognition. I haven't read "Essences", but the major criticism of it is that it was also mostly just endless examples of analogies, not delivering on the claim of the title.

Hofstader's most famous book by far was GEB, which in retrospect seems it achieved it's cult status more for it's sprawling scope and intellectual appeal rather than for any coherent exposition or explanation of its ambitious subject matter. It was a book that felt deep and clever, but really was never more than a bunch of interesting talking points, only loosely related.

Still, having read GEB I had high hopes for his "Fluid Concepts and Creative Analogies" which came 15 years later in 1995, which one might have expected to be equally ambitious in scope and breadth of ideas, but it turned out to be his least popular book, and was more of a narrow discussion on some of his GOFAI programs such as "Copycat" trying to at least demonstrate the use of analogy as a computational mechanism, if not one related to the brain.

I'm reminded now why I never bothered to read his final book "Surfaces and Essences", and reading the reviews of that confirms what anyone who had read "Fluid Concepts", or watched this video, might have suspected - that he was basically out of ideas.

For anyone interested in analogies and their role in language and thought, a book that has better stood the test of time is George Lakoff's "Metaphors We Live By".

Given that the math community (incl. those 25 Fields medalists) has come out strongly against this trophy hunting of their unsolved problems, to the detriment of mathematics, I don't think these companies are going to be getting positive press if they ignore this plea and continue with this, nor is this going to help turn public sentiment pro-AI.

Presumably the people that OpenAI and Anthropic are trying to impress with these trophy kills are potential IPO investors, but I would have thought investors would also be concerned about the growing public backlash against AI.

The result was never the point.

Clearly the real-world cannot "blow-up" - real-world water vortices do not reach infinite velocity, etc.

The point of having Navier-Stokes as a Millennium prize was to hopefully generate new mathematics and techniques along the way, and auto-generating a sprawling AI-slop proof or millions of lines of Lean does not accomplish that result.

Clearly OpenAI has no interest in the math itself - to them this was just a trophy animal to shoot and stuff. I would be very surprised if they now helped analyze the proof and try to extract the mathematical value out of it, and this would obviously require outside help who are probably not inclined to help OpenAI math-wash their behavior.

> If the worry is that AI companies are turning open problems into benchmarks and potentially “using up” fertile mathematical problems before humans can develop the ideas around them, what exactly should the companies do differently?

Just stop doing that. Don't treat unsolved math problems as some cheap benchmark to beat.

Leave the math for mathematicians, and let them use AI in a way that helps the field, not in a way that harms it.

That has to be the most impressive endorsement list any declaration has ever seen!

25 Fields medallists! Wow!