But to many commentors he's a now gatekeeping AI-hating Luddite clinging to a dying profession out of bitterness and envy because his position is more nuanced than "throw AI at everything and turn off your brain".
And yet as a society we typically consider these technologies a net positive these days, because alongside all the political instability and violence that followed them at first, some people were figuring out how to do beneficial things with them along the way that did more good than bad. Because the issue really was that the influx of more voices and ideas shifted the power dynamic, required relearning how to communicate, and in the end be better as a society.
The issue was figuring out how to "domesticate" these wild new communication channels. One successful example of which was the invention of scientific journals and papers.
Shirky predicted the internet would probably lead to a few decades of political instability too (about fifty years was his guess), and we definitely seem to be in the middle of that process.
Now, I haven't checked his stance on LLMs. I also don't know if I would quite call them a medium for mass communication like the others (as used today, they take humans out of the loop rather than let more voices join the public discourse), but I feel like they are similar enough to otherwise fit the pattern.
Tao's stance similarly feels about wanting to domesticate this wild animal before we get mauled by it. And to stick with the metaphor, I feel that most of the time the AI industry is trying to bamboozle us with spectacular rodeo displays instead.
That is probably the wildest misinterpretation of sth I have ever read here. Misalignment is used as in the goals and interests of ai COMPANIES are not the same as the ones of the mathematical community. It is very proper use of the word, and if anything imo the properest, as it refers to actual people and institutions to which actual self-ascribed goals can be defined, as in contrast to hypothetical superintelligence. AI doomers do not own some trademark on the word "alignment".
The word is being used entirely correctly, and I'm sure the nod to the AI usage is quite deliberate.
This is exactly the assumption that Tao is smuggling in with "misalignment" talk and then refusing to elaborate on any further. Is the issue that AI companies are willfully refusing to provide mathematical insight that they could provide (because they have diverging underlying "goals" to those of human mathematicians) or are they merely working under a capability gap, where current AIs can awkwardly settle major open questions but are not smart enough to provide the kind of understanding and insight that the community of human mathematicians relies on? These are two very different problems and by foregrounding the word "misalignment" in his letter so openly (as opposed to talking about AI capability to provide valued insight), Tao is picking the more adversarial reading with zero proof or motivation.
Again, nobody is accusing anyone of deliberately trying to harm mathematics. It's about misaligned objectives. Eg. Tao wrote the following before the Navier-Stokes announcement (referring to exactly the project OpenAI was undertaking):
> At this point, I would not be surprised if one could batter out such an extension by pouring an enormous amount of compute and AI assistance at such a task. But such an exercise does not particularly hold my interest; I am far more interested in digesting the proof methods and extracting out the key new insights uncovered by this approach.
And then went further to say that such activity could be actively harmful to the field. (All this can be read on Mathstodon: https://mathstodon.xyz/@tao).
The misalignment is that this activity that he and 24 other Fields medalists think is actively harmful to their field is deemed by OpenAI and others to be worth ploughing vast financial, human and compute resources into.
That's the far more sensible reading, so thanks for confirming I guess. But then the misalignment talk is pretty clearly a distraction.
> ...And then went further to say that such activity could be actively harmful to the field.
If true (and there is as of yet insufficient evidence of this), that's merely a contingent fact about very real institutional misalignment within the human mathematical community, not about AI itself or even AI frontier labs. There's simply zero inherent reason why providing a bare truth value or a completely inscrutable proof about the status of some open conjecture should make that entire subfield of math "contaminated" for the foreseeable future when it comes to extracting further human-relevant insight. That's the misalignment we should be caring about.
AI can be used to advance/deepen understanding, or it can be used to superficially go settle a whole bunch of open problems in a field without helping really in understanding them. It all depends on who uses the AI and why. Essentially, it is exactly the same concept as using the AI as a course tutor vs having it do your homework. Or using the AI to write millions of lines of code that nobody can actually read, vs keeping overview of what is going on.
In math it is probably worse because there is no objective function to maximise. Some people here think that the objective function of mathematics is to prove things, which is actually wrong. Mathematicians are not only maximising an objective function, they are also defining the objective function they need to maximise (they are defining which problems to study). The problem with AI/ML is that it can be pretty good when the goal is to maximise a set objective function, but not to set intentions and goals themselves. I would not call that a "capability gap" because we can actually get to have very useful and smart AI systems without ever reaching that point.
Though it doesn't take even an outsider to question the matching here, just look what the only guy who settled a millennium prize what he thought about the community. Or ask some actual PhD, postdoc, or even better someone who dropped along the way, how does it feel to go through this community. Not sure the goals exposed in this article are really helped with the community as it is. One again, this doesn't mean everywhere encompasses the same issues and everyone is acting badly on is own.
But if we want to accept happy pink shiny depiction of a community, we should be giving the same generosity to other communities à priori. All the more as AI and mathematical community do share a large set of common individuals.
The entire point of the paperclip maximiser is the AI isn't evil. It isn't trying to hurt humans. It just doesn't care about us.
Nobody claims OpenAI and Anthropic are out to torpedo mathematics. Just that relative to their internal goals of getting publicity ahead of IPOs by winning awards, what happens to mathematics and mathemeticians in the long run isn't a real concern.
The Holocene [1] features relatively few species humans set out to eradicate. We mostly realised something had gone extinct after we accidentally destroyed them. That is what originally misalignment meant in respect of AI.
- a rare low likelihood sequence of coincidences
- deliberate frontier lab behavior, setting intellectual interrupts on LLM "aha"-moments, so that when some mathematician explains for the umpteenth time the approach they want to take "stop paraphrasing my approach, start the actual calculation!" and when the chatbot eventually groks it, they can scoop in, possibly with live dash-board and interrupt priority levels ranging from "ensure this sample gets into the new dataset" to "Millenium Prize Scoop Opportunity Imminent, call Sam"...
There is always an alignment problem, incentives to ideals, behavior to incentives, ...