youoy

A step in the good direction, Science will definitely look similar to this.

But this guy is too high on AI, verging on AI psycosis. A more sober version is what Science will look like.

In contrast to what these philosophical pieces tend to convey, the main crisis in mathematics is a labour one.

Mathematitians exchanged proofs for employment. That was the currency. Now they cannot use that currency anymore, so the question is how does someone know if they should hire an unknow person or not?

Philosophicaly who cares if you like more theory building or theorem proving? You dont need to convince anyone. The only solution is to be curious and follow your interests and naturally humanity will redefine what the new mathematics feels like.

You cannot convince anyone about it, its not something rational.

> To take on this responsibility, we may need to broaden our view of what a mathematician can contribute. I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs.

> Our ability to understand difficult and unfamiliar ideas may become one of the most important contributions we can offer to society. We should be willing to bring that skill to problems far beyond our usual research interests. [3] Doing so asks us to expand our sense of our vocation.

Am I reading this wrong, or is he talking about what (mathematitian) Data Scientists have been doing for years? So he is basically saying that former Data Scientist that have turned into prompt/software engineers should go back to being data scientists.

In any case, people should stop trying to fit AI in the previous status quo. What we need is curious people, that is what we have always needed.

A few centuries ago there were no "mathematitians", there were mathematitians/philosophers/artists/physicists all in one person. So its not like "mathematitias" is something that has existed for millenia.

We need curious and ethical people.

Maybe AI brings back the age of a well rounded scientist/philosopher. I know this sounds counter intuitive because the article is saying that we cannot keep up with the AI.

Good point! I was speaking from my experience (at non tech companies, thats true). What i have seen is that there is an internal self selection of people that "refuse to play the game". My point is that that could have been a fair choice before, but the stakes in this new AI situation are at a point where they seem too high for that to be a good choice.

But nothing would make me more happy than knowing that there are concerned people acting internally.

Im at a point where i think the way to steer the situation we are in is to steer the machine from the inside, not quit and leave the people with the opposite mindset keep all the economic resources.

I doubt a movement of the people from the outside will work this time when all of the capital is in the other peoples hands. We need at least some of it held by someone with the opposite view to make the movement credible. And I dont think VC and IPO dynamics are compatible with this.

If you are already thinking of quiting, you can organize and internal movement and use the power of your labor to steer things. One person quitting is nothing for the company. The risk of big group quitting at the same time on the other hand... The labor is the only thing the company cares about. So we cannot wait until we are all replaceable by AI and we lose that leverage.

I agree, but I may be talking about another level of "noticing". More about observing the act and being conscious about what is happening. Thats easier it you are not rationalizing it. If you are an expert on a topic to the point where it becomes second nature, then you probably again have it easier to enter into that state, which can be a door to great creativity. But you can also get lost in all of the details that take you away from the moment.

Both cases are compatible with "noticing that you are eating a sandwich". The issue usually is that its much more difficult to "notice it" if you are thinking about how to improve it. Its like the classic parable of the camel, the needle eye and the rich. But its not impossible.

Im an ex mathematitian too. And if i was in academia I would probably have the same reaction. Thats what i say that its the worst time for proving economic value.

But if you are in for theory building and understanding, then you are not constrained anymore by your motivation to grind through countless hours of formal theorem proving. And you do not need to have superhuman formal manipulation skills and memory.

For me mathematics is not the formal system, so LLMs will never be able to do end to end maths.

Part of the controversy here is that now the skill advantage that some Field Medalist had is much narrower. The fact that fields medals have an age limit implies that it favors brain power over understanding. And that was the guiding light award of the community. So i find it "funny" (and natural) when they are offended by AI. That is the main "crisis" of mathematics.

In my opinion there has never been a better time to be a mathematitian, and there has never been a better time to be a software builder.

But there has never been a worst time to have the need to prove your economic value as a mathematitian or software developer alone. Because "understanding" is not something you can prove in one afternoon, its something that you prove with a life.

Part of the controversy here is that now the skill advantage that some Field Medalist had is much narrower. The fact that fields medals have an age limit implies that it favors brain power over understanding. And that was the guiding light award of the community. So i find it "funny" (and natural) when they are offended by AI.

That is the main "crisis" of mathematics.

In my opinion there has never been a better time to be a mathematitian, and there has never been a better time to be a software builder.

But there has never been a worst time to have the need to prove your economic value as a mathematitian or software developer alone. Because "understanding" is not something you can prove in one afternoon, its something that you prove with a life.

Thank you! We need more of this! Keep it up!

Smartphones and now AI...

I'm looking forward to Dario's essay begging for regulating the use of AI on education.

Oh wait, that goes against his economic interest.

As with the rest of the domains, AI/LLMs will do syntax and search better than any human.

In code, any developer whose differentiaton was clean code and knowledge of different technologies is now average.

In math, any mathematitian whose differentiation was to manipulate formal systems and know tricks of different domains will be average.

Fortunately, humans do more than syntax and search.

The bad news for developers is that if you know what the output of your program should be (which happens most of the time), almost all of the job is syntax and search to build the code that reproduces the output.

The good news for mathematitians is that for the majority of problems you never know the output, or you just know the output is either "True" or "False". There are some cases where you need something else, for example "a solution that blows up in finite time". For those cases AI will outperform you easily (see new Navier-Stokes solution)

So if as a mathematitian you were doing more than syntax and search, then keep doing that and use AI just for what its best.

I am not surprised that Silicon Valley researchers say they cannot solve the alignment problem... Here the leading voice in the region publishes an article about how to be more powerful without any other mention of alignment but "users".

For example, I am sure facebook users (the ones buying ads) are super happy, but is the company aligned?

Woah, better than Madoff! How can i give him money?

> The closest human parallel is self-deception, which is common and well studied by psychologists. Motivated reasoning, motivated cognition16 and the rationalizations that relieve cognitive dissonance (the discomfort of holding a belief that clashes with our actions) are all cases where thinking bends toward whatever justification suits one's interests, including one's moral self-image.

Are you describing Anthropic?

I agree. Dario is a very smart guy and he is probably aiming for the best solution for humanity that at the same time keeps VCs/whoever gives him money happy, and keeps him in control because he does not trust the rest (probably with good reasons).

My point is more that he is aiming for a local optimum for society, but he could find a lower optimum if he was not trapped in the path that he chose early on.

I dont buy the tactical nukes/end of the world narrative. Why is he not speaking about the loss of cognitive skills of the population for example? That is a more real danger that is starting to happen. There are articles already speaking about the use of LLMs as cognitive viruses. Why are his aligment teams not reasearching and publishing about this?

How can you quantitatively evaluate that? By how many times it is used in the literature? That can be gamed and there is an inherent bias on that

Will they give my auditing org access to their proprietary confidential secrets? Or will that happen only if they know i agree enough with them so that i am not a risk?

By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic. At least an open source anthropic would allow to join economic forces to try to combat that situation, for example. But there are other more clear, less distopian benefits.

Your point was that its an unfair comparison because AI its a race. My point is that bioweaponds are also a race, but a less public one. You dont have the equivalent of Dario publishing an essay every month.

Submissions are anonymous to the exterior viewers, but you do them from one account. You can extract impact metrics from that account for job applications

You mean China is not developing biological weaponds?

AI technology is now only audited by people who think like them. The ones that dont either dont join the company, or leave after a short stint as we have seen. That creates an information or feedback bubble which is not healthy nor productive.

For example I believe the world would be more secure with an open frontier AI lab, but the consequences of him doing that are too big for him.

I like to replace thes AI text with "virus manipulation"

"Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems."

If a CEO of a health company was saying this, the reactions would not be that chill.

The worst failure of our society is to call this technology AI, intead of something line "artificial general automation". The formes allows the creators to be somehow less responsible of the consequences. The later clearly moves the responsibility to the creator/user.

This sounds to me like a cry for help from someone thats held hostage.

His previous self is writting this from the possition of his current self who is too deep in the economic consequeces to be able to do anything meaningful other than write a consequenceless text. Any other action would now carry too much personal risk for him.

This all comes down to what we think mathematics is. There are two options here:

Mathematics is just the formal system: in that case we will never be able to beat the AI as humans. The goal is then to cover as much of the formal landscape with AI generated proofs to proof as much statements as possible. Success metrics are lines of lean and numbers of proven statements.

Or

The formal system is just a limited representation of what mathematics is: in that case probably some parts of the formal mathematical landscape are more important than others. Not all statements are born equal. And 90% of AI generated proofs will be just noise. Our job as mathematitians is to steer the AI to high mathematical value regions and turn formal proofs into mathematical proofs and insights.

From my point of view we have know since Godel that we are living in the reality of point 2.

Each of the two points implies a very different way of doing mathematics. So pick the one that you think is true and act accordingly.

Agree. That is also the part of the letter that does not make a lot of sense to me.

If you have taken care, then share it with the community. People will be thankful and happy.

The problem is with people that may do it without contributing to the community.