Can AI do maths yet? Thoughts from a mathematician

Why do I need to hire an artist for my movie/video game/advertisement when AI can replicate all the creativity I need.

When talking with various models of ChatGPT about research math, my biggest gripe is that it's either confidently right (10% of my work) or confidently wrong (90%). A human researcher would be right 15% of the time, unsure 50% of the time, and give helpful ideas that are right/helpful (25%) or wrong/a red herring (10%). And only 5% of the time would a good researcher be confidently wrong in a way that ChatGPT is often.

In other words, ChatGPT completely lacks the meta-layer of "having a feeling/knowing how confident it is", which is so useful in research.

I made a slight change to generalise your statement, I think you have summarised the actual marketing opportunity.

Improved tooling and techniques have given humans the free time and resources needed for arts, culture, philosophy, sports, and spending time to enjoy life! Fancy telecom technologies have allowed me to work from home and i love it :)

I find that a lot of AI+Math work is focused on the end game where you have a clear problem to solve, rather than the early exploratory work where most of the time is spent. The challenge is in making the right connections and analogies, discovering hidden useful results, asking the right questions, translating between fields.

I'm getting ready to launch [Sugaku](https://sugaku.net), where I'm trying to build tools for the above, based on processing the published math literature and training models on it. The kind of search of MR that you mentioned doing is exactly what a computer should do instead. I can create an account for you and would love some feedback.

that's if you unconditionally believe in result without any proofreading, confirmation, reproducability and even barely any details (we are given only one slide).

The "reality" of keeping this stuff secret 'cause someone would train on it is itself bizarre and certainly shouldn't be above questioning.

https://www.reddit.com/r/OpenAI/comments/1hiq4yv/comment/m30...

Has anyone had any luck with this? It seems like the only thing that it just can't do.

It often gets the actual math wrong, but it is good enough at connecting the dots between my layman's intuition and the "right answer" that I can get myself over humps that I'd previously have been hopelessly stuck on.

It does make those mistakes you're talking about very frequently, but once I'm told that the thing I'm trying to do is achievable with the Gram-Schmidt process, I can go self-educate on that further.

The big thing I've had to watch out for is that it'll usually agree that my approach is a good or valid one, even when it turns out not to be. I've learned to ask my questions in the shape of "how do I", rather than "what if I..." or "is it a good idea to...", because most of the time it'll twist itself into shapes to affirm the direction I'm taking rather than challenging and refining it.

"We might put the axioms into a reasoning apparatus like the logical machinery of Stanley Jevons, and see all geometry come out of it. That process of reasoning are replaced by symbols and formulas... may seem artificial and puerile; and it is needless to point out how disastrous it would be in teaching and how hurtful to the mental development; how deadening it would be for investigators, whose originality it would nip in the bud. But as used by Professor Hilbert, it explains and justifies itself if one remembers the end pursued." Poincare on the value of reasoning machines, but the analogy to mathematics once we have theorem-proving AI is clear (that the tools and the lie direct outputs are not the ends. Human understanding is).

"Even if such a machine produced largely incomprehensible proofs, I would imagine that we would place much less value on proofs as a goal of math. I don't think humans will stop doing mathematics... I'm not saying there will be jobs for them, but I don't think we'll stop doing math."

"Mathematics is the study of reproducible mental objects." This definition is human ("mental") and social (it implies reproducing among individuals). "Maybe in this world, mathematics would involve a broader range of inquiry... We need to renegotiate the basic goals and values of the discipline." And he gives some examples of deep questions we may tackle beyond just proving theorems.

Digression aside, my point is that I don’t think we know exactly what makes or defines “the golden hands”. And if that is the case, can we optimize for it?

Another point is that scalable fine tuning only works for verifiable stuff. Think a priori knowledge. To me that seems to be at the opposite end of the spectrum from “mess with it and see what happens”.

The best part of math (again, just for me) is that it was a journey that was done by hand with only the human intellect that computers didn't understand. The beauty of the subject was precisely that it was a journey of human intellect.

As I said elsewhere, my friends used to ask me why something was true and it was fun to explain it to them, or ask them and have them explain it to me. Now most will just use some AI.

Soulless, in my opinion. Pure mathematics should be about the art of the thing, not producing results on an assembly line like it will be with AI. Of course, the best mathematicians are going into this because it helps their current careers, not because it helps the future of the subject. Math done with AI will be a lot like Olympic running done with performance-enhancing drugs.

Yes, we will get a few more results, faster. But the results will be entirely boring.

LLMs will be tools for some math needs and even if we ever get quantum computers will be limited in what they can do.

LLMs, without pattern matching, can only do up to about integer division, and while they can calculate parity, they can't use it in their calculations.

There are several groups sitting on what are known limitations of LLMs, waiting to take advantage of those who don't understand the fundamental limitations, simplicity bias etc...

The hype will meet reality soon and we will figure out where they work and where they are problematic over the next few years.

But even the most celebrated achievements like proof finding with Lean, heavily depends on smart people producing hints that machines can use.

Basically lots of the fundamental hints of the limits of computation still hold.

Model logic may be an accessable way to approach the limits of statistical inference if you want to know one path yourself.

A lot of what is in this article relates to some the known fundamental limitations.

Remember that for all the amazing progress, one of the core founders of the perceptron, Pitts drank him self to death in the 50s because it was shown that they were insufficient to accurately model biological neurons.

Optimism is high, but reality will hit soon.

So think of it as new tools that will be available to your child, not a replacement.

Every LLM release moves half of the remaining way to the minimum viable goal of replacing a third class undergrad. If your business or research initiative is fine with that level of competence then you will find utility.

The problem is that I don't know anyone who would find that useful. Nor does it fit within any existing working methodology we have. And on top of that the verification of any output can take considerably longer than just doing it yourself in the first place, particularly where it goes off the rails, which it does all the time. I mean it was 3 months ago I was arguing with a model over it not understanding place-value systems properly, something we teach 7 year olds here?

But the abstract problem is at a higher level. If it doesn't become a general utility for people outside of mathematics, which is very very evident at the moment by the poor overall adoption and very public criticism of the poor result quality, then the funding will dry up. Models cost lots of money to train and if you don't have customers it's not happening and no one is going to lend you the money any more. And then it's moot.

This time could be different, of course. But I'll need a lot more evidence before I start telling people to base their major life decisions on projected technological change.

That's before we even consider that only a very slim minority of the people who study math (or physics or statistics or biology or literature or...) go on to work in the field of math (or physics or statistics or biology or literature or...). AI could completely take over math research and still have next to impact on the value of the skills one acquires from studying math.

Or if you want to be more fatalistic about it: if AI is going to put everyone out of work then it doesn't really matter what you do now to prepare for it. Might as well follow your interests in the meantime.

Most likely AI will be good at some things and not others, and mathematicians will just move to whatever AI isn't good at.

Alternatively, if AI is able to do all math at a level above PhDs, then its going to be a brave new world and basically the singularity. Everything will change so much that speculating about it will probably be useless.

(｡•́︿•̀｡)

Evaluating the output of llms will also require mathematical skills.

But I'd go further, if your son enjoys mathematics and has some ability in the area, it's wonderful for your inner life. Anyone who becomes sufficiently interested in anything will rediscover mathematics lurking at the bottom.

There's never been the equivalent of the 'bench scientist' in mathematics and there aren't many direct careers in mathematics, or pure mathematics at least - so very few people ultimately become researchers. Instead, I think you take your way of thinking and apply it to whatever else you do (and it certainly doesn't do any harm to understand various mathematical concepts incredibly well).

But specifically to your worry about humans just dotting i's and crossing t's, he predicts that exactly the opposite will happen. At the end he emphasizes that the ultimate goal of mathematics is more about human understanding than proving theorems.

The universal approximation theorem. And that's basically it. The rest is empirical.

No matter which physical processes happen inside the human brain, a sufficiently large neural network can approximate them. Barring unknowns like super-Turing computational processes in the brain.

Or perhaps, determinism and mechanistic materialism - which in STEM-adjacent circles has a relatively prevalent adherence.

Worldviews which strip a human being of agency in the sense you invoke crop up quite a lot today in such spaces. If you start of adopting a view like this, you have a deflationary sword which can cut down most any notion that's not mechanistic in terms of mechanistic parts. "Meaning? Well that's just an emergent phenomenon of the influence of such and such causal factors in the unrolling of a deterministic physical system."

Similar for reasoning, etc.

Now obviously large swathes of people don't really subscribe to this - but it is prevalent and ties in well with utopian progress stories. If something is amenable to mechanistic dissection, possibly it's amenable to mechanistic control. And that's what our education is really good at teaching us. So such stories end up having intoxicating "hype" effects and drive fundraising, and so we get where we are.

For one, I wish people were just excited about making computers do things they couldn't do before, without needing to dress it up as something more than it is. "This model can prove a set of theorems in this format with such and such limits and efficiency"

It's worth considering why "everyone seems all too ready to make ... leaps ..." "Neural", "intelligence", "learning", and others are metaphors that have performed very well as marketing slogans. Behind the marketing slogans are deep-pocketed, platformed corporate and government (i.e. socio-rational collective) interests. Educational institutions (another socio-rational collective) and their leaders have on the whole postured as trainers and preparers for the "real world" (i.e. a job), which means they accept, support, and promote the corporate narratives about techno-utopia. Which institutions are left to check the narratives? Who has time to ask questions given the need to learn all the technobabble (by paying hundreds of thousands for 120 university credits) to become a competitive job candidate?

I've found there are many voices speaking against the hype---indeed, even (rightly) questioning the epistemic underpinnings of AI. But they're ignored and out-shouted by tech marketing, fundraising politicians, and engagement-driven media.

Since chatgpt-4o, there has been o1-preview, and o1 (full) is out. They just announced o3 got a 25% on frontiermath which is what this article is a reaction to. So, any tests on 4o are at least TWO (or three) AI releases with new capabilities.

https://news.ycombinator.com/item?id=42473470

This is factually wrong. The most interesting problems motivating the quantum computing research are hard to solve, but easy to verify on classical computers. The factorization problem is the most classical example.

The problem is that existing quantum computers are not powerful enough to solve the interesting problems, so researchers have to invent semi-artificial problems to demonstrate "quantum advantage" to keep the funding flowing.

There is a plethora of opportunities for LLMs to show their worth. For example, finding interesting links between different areas of research or being a proof assistant in a math/programming formal verification system. There is a lot of ongoing work in this area, but at the moment signal-to-noise ratio of such tools is too low for them to be practical.

My understanding is that many problems have solutions that are easier to verify than to solve using classical computing. e.g. prime factorization

How do you verify what a particular open model was trained on if you haven’t trained it yourself? Typically, for open models, you only get the architecture and the trained weights. How can you reliably verify what the model was trained on from this?

Even if they provide the training set (which is not typically the case), you still have to take their word for it—that’s not really "verification."

This is a correct statement about the worst case runtime. What is interesting for practical applications is whether such statements are among those that you are practically interested in.

If every true theorem had a proof in a computationally bounded length the halting problem would be solvable. So the AI can't find some of those proofs.

The reason I say 3 is deep is that ultimately our foundational reasons to assume ZFC+the bits we need for logic come from philosohical groundings and not everyone accepts the same ones. Ultrafinitists and large cardinal theorists are both kinds of people I've met.

This has little to do with the usefulness of LLMs for research-level mathematics though. I do not think that anyone is hoping to get a decision procedure out of it, but rather something that would imitate human reasoning, which is heavily based on analogies ("we want to solve this problem, which shares some similarities with that other solved problem, can we apply the same proof strategy? if not, can we generalise the strategy so that it becomes applicable?").

Why do you say this? The AI doesn't know or care about soundness. Probably it has mathematical intuition that makes unsound assumptions, like human mathematicians do.

> How likely are each of these outcomes?

I think they'll all be true to a certain extent, just as they are for human mathematicians. There will probably be certain classes of extremely long proofs that the AI has no trouble discovering (because they have some kind of structure, just not structure that can be expressed in ZFC), certain truths that the AI makes an intuitive leap to despite not being able to prove them in ZFC (just as human mathematicians do), and certain short statements that the AI cannot prove one way or another (like Goldbach or twin primes or what have you, again, just as human mathematicians can't).

Previously workers in a field disrupted by automation would retrain to a different part of the economy.

If AI pans out to the point that there are mass layoffs in hundreds of sectors of the economy at once, then i’m not sure the process we have haphazardly set up now will work. People will have no idea where to go beyond manual labor. (But this will be difficult due to the obesity crisis - but maybe it will save lives in a weird way).

> programmers are still in the denial phase

I am doing a startup and would jump on any way to make the development or process more efficient. But the only thing LLMs are really good for are investor pitches.

Which is actually a problem I have with ARC (and IQ tests more generally): it is computationally cheaper to go from ARC transformation rule -> ARC problem than it is the other way around. But this means it’s pretty easy to generate ARC problems with non-unique solutions.

Edit: I do see from your profile that you are a real person though, so I say this with more respect.

You won't get the Turing machine evaluation mechanism and determinism, but you will have a generator. Although the viability of what is generated is is question. Because the other part of formalism, semantics, is almost always missing.

No "AI" of any description is doing novel proofs at the moment. Not o3, or anything else.

LLMs are good for chatting about basic intuition with, up to and including complex subjects, if and only if there are publically available data on the topic which have been fed to the LLM during its training. They're good at doing summaries and overviews of specific things (if you push them around and insist they don't waffle and ignore garbage carefully and keep your critical thinking hat on, etc etc).

It's like having a magnifying glass that focuses in on the small little maths question you might have, without you having to sift through ten blogs or videos or whatever.

That's hardly going to replace graduate students doing proofs with professors, though, at least not with the methods being employed thus far!

So, for any LLM, if you intersperse more than that number of ‘X’ tokens between each useful token, they won’t be able to do anything resembling intelligence.

The current LLMs are a bit like n-gram databases that do not use letters, but larger units.

the management class has a strong incentive to believe in this narrative, since it helps them reduce labor cost. so they are investing in it.

eventually, the emperor will be seen to have no clothes at least in some usecases for which it is being peddled right now.

This might be a dogma that needs to die.

AI doesn't need to learn everything, our LLM Models already contain EVERYTHING. Including ways of how to find a solution step by step.

Which means, you can tell an LLM to translate whatever you want, into a logical language and use an external logic verifier. The only thing a LLM or AI needs to 'understand' at this point is to make sure that the statistical translation from left to right is high enough.

Your brain doesn't just do logic out of the box, You conclude things and formulate them.

And plenty of companies work on this. Its the same with programming, if you are able to write code and execute it, you execute it until the compiler errors are gone. Now your LLM can write valid code out of the box. Let the LLM write unit tests, now it can verify itself.

Claude for example offers you, out of the box, to write a validation script. You can give claude back the output of the script claude suggested to you.

Don't underestimate LLMs

The "open worm project" was an effort years ago to get computer scientists involved in trying to understand what "software" a very small actual brain could run. I believe progress here has been very slow and that an idea of ignorance that much larger brains involve.

https://en.wikipedia.org/wiki/OpenWorm

https://mathstodon.xyz/@tao/113132502735585408

Very many things conventionally labelled in the 50's.

You are speaking of LLMs.