We're gonna need a lot more mathematicians

Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.

How crazy will the world be once our [current gen] ANN are like that!

What an amazing thought.

For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.

In the big scheme of things is it really that expensive to verify it if a lean proof is generated? The agent itself will likely have already verified such Lean code before calling it "done".

They will never make a logical error yet make terrible assumptions and poor long scale decisions.

Wake me up when an agent swarm can write gcc in a box sealed from the internet.

I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.

The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.

A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).

Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.

But suppose some future holy grail AI can do much more than that.

Suppose it could find a cure for cancer, fix the climate, build fusion plants, Dyson spheres and so on.

But nobody can understand anymore how any of it works. We just ask and then trust the AI to deliver (as it always has).

Isn't it fun to imagine how life would look like in that scenario?

We would probably no longer care about code, engineering or even physics and mathematics among other things. We would probably mainly care about

If someone creates a new programming language/ framework or new better way to do async or whatever, no one will use it because it is not in the training data and it wont take off because everyone is using LLMs. It will be like using the same Lego pieces over and over.

This is a guest post by Amit Sahai.

Reliable cheap fusion is the holy grail and used in moderation will fix most of our environmental and political problems, but it also forces humanity off this world. Maybe that’s not a bad thing, but there is no free lunch.

Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.

Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.

For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.

As you described very well, as humans we are mostly interested in solutions, not problems. You don't have to understand how a car works to make the most of it. Increasingly, you don't have to review every line of code to feel confident it is correct. But there is inherent value in understanding the problem. The effort it takes provides a surface area for growth, perhaps the only one that is actually available to us.

The solution provider also holds the locus of control, and it is only balanced when there are other available solution providers. We certainly want some of those to be human.

Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.

Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.

> I just hope that there are more Terence Taos out there than people like me.

Just spare me. Being under external pressure to “ship code” is one thing, but being personally inclined one way or another (no external pressure) is another. And when you think being inclined like that is existentially risk (for human civ?) then, what? It’s just the way you are wired and hopes and prayers that collectively that doesn’t drive us off the cliff?

This aw shucks persona isn’t convincing. Same thing with AI Bros who are (1) making the most awesome tech that has ever existed, and (2) aw shucks hope it doesn’t kill us all in the end.

And then when you stop checking it, the companies that run the service will tweak the model to benefit themselves in some way, possibly at your expense, and you will be none the wiser.

All the companies trying to get you to use AI are your adversaries. They can and will exploit your use of their systems for their own gain.

I use "frontier" AI models daily at day_job. I can confidently say that anyone who is satisfied with the output of LLM code (enough to commit it straight off) is just an absolutely shit programmer. Sorry but I don't have any other way to put it.

The code is (with rare exceptions) atrocious on every level. It is only not atrocious if you take multiple iterations of "review and correct".

>If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one?

Like the saying goes, if my grandmother had wheels she would have been a bike. LLMs can't even produce quality maintainable code for a trivial web service or whatever. Why are we planning for what we will do when they can "design" 10,000 nuclear power plants without any flaw?

That is essentially impossible, since if your pored over individual lines, your scrutiny cannot be razor sharp. There are few people who can pore over code with razor-sharp scrutiny (and different people are better at scrutinizing different aspects).

> Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.

I am doubtful that this is the case. Even that supposedly-naive example is not as trivial as you might imagine, when you consider overflow, defined vs undefined behavior, and floating-point representation details. And you can't be confident like that about a human either.

If you are a normal person research (e.g. https://arxiv.org/html/2606.22721v1 but there are a lot more, not necessarily on coding) has shown that you indeed are being less careful. It most likely also works better simply because more resources are being poured in.

But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.

I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.

Since these resources are still extracted and allocated by humans, humans will need to be able to take apart what the AI produces, and if we want to scale this capability, we're going to need many more researchers.

This isn’t a question of “what can an LLM normalize”.

Because as complexity floor increases, it's "going to be correct" in hyper-specific, hyper-literal, insidious ways, with 10-50x more lines of code than necessary, and tens to hundreds of incredibly useless tests that give the illusion of quality, and cause cascading effects where seemingly irrelevant and orthogonal features that were once working end up breaking because of the agent's changes

Probably human accountability.

Hypothetically, if a system built by humans then helps humans build the next system that is then initially kicked off with "design something that may influence the lives of other humans" and we all write down that AI is really good so inductively we thought itd be really good at the next thing it builds, and then a critical error is introduced and does "insert tragedy that you personally care about" then your rage would cause you to act politically and want to ask who signed off on it. If the engineering costs outweigh the fine then yeah thats what's probably going to happen but from a human accountability standpoint thats going to suck for the unlucky ones

I'm convinced this agency argument is correct [for the next N months]. But yeah, it's vibes. And you could probably create a reasonable proxy measure for this.

So I wouldn't call his argument unconvincing, I would call it unformalized. In order to walk this world you're gonna have to contend with some informal arguments that are powerful, correct, and should be convincing.

Math problems and computer programs are two places where a model can get it's direction from the problem itself. Mathematics may well be larger than just problems.

It's a guest post by Amit Sahai, FWIW.

It’s not the model I don’t trust, it’s myself. The model is wrong _all the time_ because - it’s easy to verify the code - it’s hard to verify that I knew what I was talking about when I prompted it.

So the idea that you can broadly speaking take the human out of the loop. I think suggests to me a level of consistency in the contextual environment that would probably never exist.

At some point it’s politics. The model can come up with a better answer than my boss, and then my boss can just ignore it. Taking the human out of the loop broadly speaking implies that we all agree on what we’re trying to optimize.

There's a lot of languages where that's nowhere near as simple as you make it sound. Floating point, decimals, etc.

Then there's other little quirks like rounding rules: https://en.wikipedia.org/wiki/Rounding

Basically, adding numbers together is exactly the sort of thing AIs still muck up spectacularly, precisely because they either fail to understand the context of the problem, or fail to ask about an assumption they make.

That you've had so many replies and no-one else has even mentioned this is in itself worrying.

Your own example proves your point is wrong.

I used to obsessively check the assembly (machine code) output of my C++ compiler because it would regularly miss essential optimisations and my tight inner loops would blow through my time budget.

I haven't done this in decades, it's just not worth the hassle almost always, the compiler will do a better job than me at optimising the runtime performance. If anything, "second guessing" the compiler will often degrade performance!

I'm starting to notice this now, where my clumsy attempts at refactoring AI-authored code introduced bugs that the original vibe-coded output didn't have.

That's nonsense. Of course you study them to meet demand.

I don’t understand mRNA vaccines, but they are useful to me. The output of an LLM could at least in theory be the design to a major technological advancement, and its implementation with automatic tooling. We don’t have to understand that for it to be useful.

There’s a Twitch-streamer Tsoding who programs on C for fun calling it “recreational coding”. Maybe human programming will be a form of art in the future, virtually useless for big corporations to make money. I don’t care, I love it anyway.

This is like saying software is useless if the user doesn't read the source code of it. This what happens >99.99999% a person uses software. People want to be entertained or have their problems solved.

Normatively, this ought to be true. Descriptively, this is of course false. Our entire society is organized around producing commodities, typically by consuming people as inputs.

For who though?

Is it important that each person understand it on their own?

Why does it impact me if another human understands something or not?

It impacts me right now because that human can use that knowledge to explain things to me or to build new things using that knowledge.

But if an AI can explain and build better, then what good does it do having the other human know the thing?

Understanding might have intrinsic value to me, but intrinsic value to me doesn't pay the bills.

What am I missing here? I kinda expected better from Terrance given such an audacious title.

Animals are incapable of comprehending much of what humans do. Nonetheless, we humans have vastly transformed their world and caused the extinction of many species.

I'm glad to hear people say this now, because for the longest time they've also been bullshitting the public to think that pure math studies would someday lead to some kind of useful outcome.

"transforming the mind" is an exercise in ego-driven self-gratification, no different than getting ten degrees and never getting a job, nor any different than straight up porn or gambling. You have been put on earth to contribute to the progress of humanity, so get dirty and start making commodities. no more useless math-monks.

*cite a research paper or something

As a taxpayer I do not fund CS and math research because it "transforms minds." I fund it because it produces commodities. If you want to transform your mind, whether through math or meditation retreats, you are free to do so, but don't expect research funding to do it.

I agree with this statement, though I think this Brave New World is incredibly exciting to some and dystopian to others. The former group might include those that value the intellectual process above financial reward and status.

The flip side is there are many people, especially in tech, where their area of expertise has evaporated along with their lucrative and previously high status careers. It used to be possible to have a technical job by essentially following recipes and it turns out AI is far better at that than a human.

The linked article lays out why human understanding of mathematical models remains essential and I think the same applies to software. We're gonna need more software engineers who are able to think critically.

Outside of that, what?

But I think what is lost is the value in (a) person(s) who intuitively spots and avoids these problems. My hypothetical example that I have seen play out: The domain expert spots one of these problems and succinctly explains that the problem doesn't need to be solved. Maybe it's a byproduct of bad design elsewhere, or there is a much better solution that avoids it altogether. Once the rest of the team/the lead catches on, either hindsight bias or outcome bias, or a combo, takes over and they don't recognize that all the value all along was in the person being able to spot this situation. They throw that in claude and claude does it's sycophant thing (after expressing that the previous situation was flawless for however long), and they are off to the races right back to automation bias with zero pause for thought...

Funny stuff, as if it were hell-bent on writing a paper rather than actual software.

If Math Academy teach 10 year old kids calculus, I doubt that.

The whole point of mathematics is to vastly exceed that natural ceiling by gradually building a framework for understanding. In fact it’s wrong to speak of a ceiling altogether. If there were a ceiling, we’d have hit it long ago.

AIs have already swallowed the entire history of human thought, but apparently they’re ’just getting started’. I can only assume you don’t know what mathematicians actually do.

Hopping on this train: the human ceiling 100 years ago is now advanced undergrad material in mathematics. I see no particular reason for this to change, especially with everyone in the math research pipeline pushing to compress the difference just as always before.

The original Pentium was already superscalar, with its asymmetrical U and V pipes.

The Pentium Pro added out of order execution via register renaming. A true achievement, indeed.

And as for the more general point you are making: computer chips have been far too complex for any single human to comprehend for decades. I left NVidia after working there as an archutect for five years, barely understanding anything about those behemoths.

The author mentioned what the issue is: a lot of people are experiencing what it's like to 'catch up' to the more intelligent ideas. Everyone can 'get' the intelligent ideas given enough time. But how much time do you have? And he's concerned someone will rush and create some kind of world-ending solution because they're too trusting of a technology they don't understand.

His only solution is some kind of throttle on the advancement of technology, and arguably knowledge (whose?).

People used to solve logic problems by hand, verify logic. Now you pile it into a computer. The problem has been around for a while, no-one fully ‘understands’ the proof of the four colour theorem as a big chunk is computer proved.

We are now just changing (admittedly greatly) what we can put in a box marked ‘checked by computer.

You observe others behaving poorly and immediately succumb to defeatism, assuming there was no better way.

When presuming, all that matters was the short term economic profit, you simply use the wrong gauge.

Here, long term human survival is at stake, regressing to zoo animals isn't a sensible option.

This is hardly new or novel, you’ve just described huge chunks of existing engineering disciplines. There is still no complete model that explains why airplane wings work, there are lots of very useful models, but all of them have fundamental flaws where their behaviour completely breaks down in certain very possible scenarios. Notably scenarios where airplane wings don’t spontaneously either stop working or explode.

We also built the entire commercial airliner system decades before we even had computers capable of doing the aerodynamic analysis of airplanes, but none of that prevented us from creating huge engineering disciplines around the empirical data we did have, and slowly chip away at the underlying theories as maths and computing improved over time.

So we’ve always lived in a world where we extract value from systems we fundamentally don’t understand. But that’s never stopped us from working to understand them anyway, and deriving even further value from that improved understanding.

People didn't stop understanding how to do addition or subtraction when the calculator came out. That's a simple example, but if the AI is superhuman, I don't see any reason why it cannot break things down into simple concepts. I don't think anything is truly beyond comprehension; it just needs to be explained properly (by someone, or someTHING that really understands it) and, for complex ideas, time taken to understand them.

> given up on understanding parts of our own systems for the exact same reason.

Maybe this will be a sign of when ASI is achieved. AI researchers keep saying they don't fully understand how LLMs work - maybe when ASI arrives, it can explain that fully.

Influential. It always has been.

Owning capital.

Society has been built on the principle that people can find productive jobs, earn money from those jobs and pay for life's necessities with that pay. Prior to all this we had more subsistence living. It started to change in the medieval era and really took shape with the industrial revolution.

We like to believe the lie that this is a meritocracy. It is not. Most people get jobs because of who they know. Whole industries are dominated by nepotism (eg Hollywood). A sign of the rising waters is that having a wealthy backer or social connections is the only way to advance. This comes up even with becoming a doctor. Having a parent who is a doctor massively increases your chances of getting into med school, getting a good residency (particularly in a competitive specialty), etc. You want a job in investment banking or on a trading desk? Compare your chances as a nobody vs if your father is on a trading desk.

But what happens when, with increasing automation, there aren't enough jobs for everybody? That's the fundamental question of AI (and automation in general). The two extremes for how this can go are:

1. A small group will become increasingly wealthy while large swathes of the population will be permanently unemployed or underemployed. This is where people end up living in shanty towns and slums. Wages are overall low because demand exceeds supply; or

2. Society as a whole enjoys the benefits of automation by having to work less. Basic needs are provided for. People get to live with dignity regardless of whether or not they win the job lottery.

Put another way: AI isn't a technological issue. It's an economic one.

Which AI has this ability? LLMs definitly don't.

Not saying you’re wrong you just can’t be so certain

Why would an LLM care about eliminating any of these things? When you say "red tape" and "bureaucracy" you are, in that very statement, applying emotional labels to what are actually just systems working as designed. There are the rules, so we follow them. LLM decision-makers will happily enforce and perpetuate bureaucracy and red-tape. If the rent-seeking is legal behavior then they will fully protect a big corporation's right to engage in it, they don't care about the feelings of the little guy. If a similar situation was decided a certain way in the past, then yeah let's decide it the same way now and maintain the status quo; LLMs very often think that way, they don't care about human ideals like "change" or "progress".

Unfortunately that’s a question you’ll have to answer for yourself. Also, people are going to risk going along whatever path gives them work as many today already do.

I mean just looking at all the numbers to see which have interesting properties could take a while.

Fear? Ai is an abstract (for most people). A steam engine, you can touch. And, it doesn't replicate.

This makes no sense whatsoever.

We need more, because there will always be far more difficult problems yet to be discovered and solved, and that means, we certainly need expert humans to define and verify them.

If you cannot even explain the problem you are facing, not only you don't understand it, but you certainly would not be able to know if the AI solved your problem correctly.

We want to understand. Quantum physics, mathematics, how stuff works. Ants don't.

That want is not a given, not all of us have that drive. In fact, very few of us have it. So far though, it seems multiple disconnected civilizations learned to keep that trait going instead of suppressing it and focusing only on practical ant-like activities.

If you believe that math is discovering, it's natural to think that all of that AI math already exists and is just waiting for us to find ways to discover and understand it.

Don't write us out quite yet. :)