I'm not a mathematician either, but I find it hard to believe we could run out of problems in pure mathematics. Pure math is like a game where if you get bored you can invent new rules. Maybe eventually all fields related to applied problems will be settled, and the value of pure mathematics will be relegated to the status of other games like chess and go. But that's so far away from current reality that I can't imagine it happening in the next century at least.
Frontier labs aren't under any obligation to release every model they develop. Either way, I don't see much evidence that they're withholding them in perpetuity. Open models aren't far behind so there's no incentive for that.
Difficult math problems are useful benchmarks and milestones. It's well worth throwing money at solving a problem if it drives competition and improves models significantly. The benefits become available to everyone, including the thousands of professional mathematicians who can use them to become more productive.
This. I've recently (re-)discovered the importance of using writing to force myself to confront my assumptions and gaps in knowledge. It's easy to fool yourself into thinking that you understand something if you haven't made it concrete and explicit by exercising your own brain. Arguably, this extends beyond writing in the conventional sense to many cognitive domains: mathematics, programming, physics, engineering, etc. AI should help the process of thinking, not replace it.
George Orwell's essays. I love his simple, matter-of-fact prose style and I'm trying to absorb as much of it as possible to improve my own writing. Many of them are available online if you don't want a full book. "Politics and the English Language" is a must-read.
I just asked ChatGPT to multiply two 4-digit numbers, and two 7-digit numbers without external help. It got both right. I'm sure it wouldn't have a 100% success rate, but saying it can't do arithmetic is just false.
I wouldn't say that information is an upper bound on knowledge because we don't measure knowledge in bits. The number of possible sequences of N bits is 2^N and knowledge involves selecting the sequences that are useful in some way. I don't know how to quantify it, but in principle it could be much larger than N.
I don't think I agree with your characterization of the second definition. Time scales matter. It's not much use to be able to solve human-scale problems if it takes millennia. And it only takes months to train an LLM to the level that it can solve cutting-edge math problems.
I like to think of modern pixel art as its own aesthetic and art form. It's designed to be viewed on high DPI LED screens, and that's fine. I don't think it should be judged by how it looks on a CRT.
There's an undeniable charm to the older era of pixel art, and surely part of that is nostalgia, but I also think a lot of it comes from the fact that the economics and the hardware were a lot different. Up until the mid 1990s, the highest paid artists in the industry were working on pixel art games. The hardware was slow enough that there were real creative constraints, and we know that constraints often drive creativity.
Note I used the word knowledge, not information. A random string can also contain 1 gigabyte of information.
Imagine an alien that matches your abilities across every domain, but has a 10 billion year training period, something many orders of magnitude more expensive than an LLM. I simply don't believe that alien is less intelligent than you.
We also don't expect humans to be competent in every domain. Most humans suck at most things. We will usually call someone intelligent if they excel at solving problems in one or two narrow domains.
I think you're underestimating how much knowledge about the world is encoded in human DNA, especially in the structure of the human brain at birth. It also depends how we count the "operations" used to train a human adult, even if we ignore the evolutionary history.
I'm still going to deny the premise of your argument, becasue I think we should define intelligence in terms of capabilities. If a system can discover a cure for cancer or solve P vs. NP, it doesn't matter how many FLOPs it took to train.
If we're including the training process and not just the final product, why shouldn't we include the billions of years of natural selection encoded in DNA sequences?
Intelligence is a word we invent to describe things we see in nature. We don't "discover" intelligence like it's some natural resource. To say we know nothing about it is also a bit strange. Cognitive science has been studying it for decades. Of course it's hard to give a precise definition, but it's related to capabilities like abstraction, reasoning, planning, problem solving, etc.
Are they denying intelligence, or are they redefining it in such a way that only humans can be intelligent? Can you come up with a definition of intelligence that would apply to crows and ant colonies, which are obviously intelligent to some degree, but not the current generation of AI systems?
It's revealing that the author thinks that a podcast is the ideal venue to reason through complex technical issues.
The "like" phenomenon is less of a verbal fluency issue and more of a cultural one. You can listen to someone smart like Donald Knuth speak and he will definitely stammer and stutter a lot, but he doesn't use filler words. "Like" is more of a West Coast or California thing and I also find it annoying and distracting.
"Burning academia to the ground" is a terrible idea. We need to fix funding and incentives. If funding and research all happen in industry, where are the incentives to do basic research?
Your analysis completely ignores the physical and biological sciences, engineering, and the humanities. It mostly applies to a small subset of academic fields in the social sciences and medicine. You're also ignoring the changes that have happened since the replication crisis. Preregistration, publishing all code and data, reporting null findings, replicating results, etc. are becoming the norm.
Academics themselves rarely frame their research in sensational terms. This happens in press releases and media coverage, which they have little control over. You can see this by comparing the title of the press release to the quotes by the actual researchers:
"We’ve shown for the first time that the front of the brain arises from a totally different progenitor cell than the back of the brain."
"I was surprised at our findings because the word ‘brain’ implies a contiguous organ that likely has a singular origin," Jokhai said. "But even 500 million years ago, there were these separate neural systems, which now almost operate as one, which is very cool."
I don't hear much overselling or hype in this, I hear an accurate description of the research and a bit of nerding out over it.
That's simply not true. I've spent hundreds of hours listening to his AMAs and he's remarkably good at taking questions that might be confused or poorly worded and respectfully answering the best interpretation of what the person asked.
He does have an approach to certain questions to say that it's the wrong question to ask. For example, to "why does the universe exist?" he would say that the question presupposes that everything needs an explanation for why it exists, and the universe is not that kind of thing. This is actually super common in philosophy, but I could see it coming across as pompous to someone who had never heard an answer like that before. He's not saying that the question is stupid, he's saying that it literally doesn't have an answer.
It's the AI doomers who have chosen to frame the argument in terms of extinction. Even if they were only claiming that 1% of the world population will be killed in the next 10 years, it would still be an extraordinary claim that hasn't been established with any amount of rigor. I'll reiterate the original point: these people are not serious, they're spreading alarmist nonsense, and they're undermining the public's trust in experts.
None of those are arguments giving a probability of human extinction before 2036 or any other time. AI Futures has said that some members of their team believe that it's 10-30% at some unspecified point in the future, but there is no justification given for where those numbers come from.
Mindscape is roughly in the top 20 science podcasts (also #3 in physics) and his books regularly make the NYT bestseller list. So he's definitely popular, but perhaps still underrated relative to the quality of his content. His approach is more idiosyncratic than the most popular science communicators. He focuses mostly on the topics that interest him and isn't averse to getting technical in certain discussions. I think that naturally limits his audience, for better or for worse.
None of that has anything to do with the article, and if you thought the message was "this is overhyped bullshit" then you should go back and read it again. As I already pointed out, he isn't saying anything about the probablity of harms or disasters from AI. He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence.
Those books are irresponsible too, but the authors aren't experts on what they're writing about. This 10% claim comes from employees at Anthropic. That's the whole point of the article: expert opinion carries weight, fear is contagious, and people have extreme reactions to extreme claims.
This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.
I like Gaben's take on going public. This is a clip from a longer talk he gave which I think is really insightful into many aspects of the economics of software and company culture.
The American Revolution was very much a discussion about power. Of course, it wasn't just a discussion (we had to fight a war to defend the new form of government), but it was driven by the same concerns about the distribution of power.
I don't agree that the argument implies nationalization. Regulation would work if it slowed everyone down at the same rate, enough to mitigate the risks. The problem is that you need regulators who know what they're doing and strong international cooperation. If you regulate a fraction of the global market, you just create an incentive to shift development to other countries. Nationalization, being essentially the most heavy-handed form of regulation, faces the same problems and comes with its own risks as well.
We're talking about this like all of the bad incentives are created by market competition, but that's not really the case. Most of the incentives come from untapped value in the form of potential profits, strategic advantage, military superiority, etc. Corporations and governments want to capture this value for themselves, creating various types of competition. Dario's argument depends on the assumption that the primary risk comes from the pace of development and threats from the technology itself. That's probably where I disagree the most; I think the highest risk is rising authoritarianism and competition between nation states. Slowing down isn't really a solution to those problems.
The article is about muscle hypertrophy, not health outcomes.
>if you’ve only got an hour a day or whatever for a workout you’re better off spending it all on resistance training.
The health benefits of resistance training plateau around 90-120 minutes per week. And don't forget about cardio, which is arguably more important for reducing disease risk. I would guess that maintaining flexibility helps reduce risk of injury and mortality from falling, which becomes increasingly likely as you get older. But there's not much evidence that increasing flexibility beyond a normal functioning range has any ancillary benefits.
I agree with your analogy, but I don't understand the last part. If I'm working on a hobby project, say a video game, the "tomatoes" I get are the experiences of writing code, playing a level I designed, and sharing it with my friends. The advent of AI doesn't reduce the value of those experiences or make them go away.