Pacing the frontier
pacingthefrontier.com
pacingthefrontier.com
I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? We should want the AI Labs to make binding agreements to proceed safely and slower, rather than being stuck in a suboptimal game theoretical race to advance unsafely. We should be hoping that nations can make binding agreements. Our discourse here should be about the substance of what these agreements could be and the practicality of enforcement mechanisms.
If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?
I wouldn't frame it as moving the goalposts in a bad way, though. We understand more now than we did then, so we have a better idea of what the actual goal is, and moving the goalposts to frame that goal is more rational than insisting the goalposts stay where they were ten years ago even though the entire pitch has changed since then. I'm straining the analogy, obviously.
Note well that I have not said LLMs are not useful, only that no amount of interaction with them has convinced me they remotely approach general intelligence. Their actual function of heuristically regurgitating all information ever known to mankind remains extremely useful in lots of domains.
And by the way, it's not obvious an LLM could fool me (or the average person) over a sufficiently long period of time, exactly because they are heuristic machines. That they lack thought guided by underlying cognitive models seems to always leak out, in the end. I suspect pass rates by LLMs on long-range Turing tests (if there have been any conducted) might drop as people become acclimatized to them.
Which takes priority: public safety or the free-market + national security? The AI labs hype in now biting them in the ass: why would China and the rest of the world deny their businesses a technology that "is bigger than the invention of electricity" or whatever hyperbole the snake-oil salesmen came up with when they were not breathlessly declaring that the first side to get AGI wins forever.
Even if the pacers are right, it's like greenhouse emissions all over again; each country wants everyone else to cut emissions, but no one wants the economic hit of doing so.
And to your point. Don Quixote, the first modern and greatest novel of all times, is about a guy that lost touch with reality from reading too many books
The point is, it should give you at least some pause when the people most familiar with a technology are among those making the strongest statements about its potential dangers. Extrapolation has worked extremely well in predicting what AI will be capable of over the past 5 years; how confident are you that it will fail before we reach apocalyptic scenarios?
People at anthropic don’t have any secret that many people outside already have. Again the burden of proof is on the person making a claim. That you feel concerned or scared doesn’t grant you special privileges to control or dictate a transforming technology.
The motivations aren't unique here. Restrictions on the development of almost any technology benefits incumbents by providing a moat.
If your only evidence that the claim "ai dangerous - regulate ai" is true is "insiders who stand to make billions if they can regulate away the competition" told me so, that is pretty weak evidence indeed.
If a group of concerned scientists (without $$$ in stock options) demonstrated that anyone with one year of HS chemistry, an Amazon account, and a ChatGPT max plan could build a bioweapon 10x more lethal than an AR15 mass shooting or driving a truck into a crowd (because it is certainly more than 10x harder to make a bioweapon than either of these things, even with a theoretical AGI), then that would be worrisome. This is really not that high of an evidence bar but even it has not been met, but oh no we should start regulating yesterday.
On the other hand, I'm pessimistic about initiatives like this being effective on a global scale (though they could certainly work at the national level).
For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
The other justifications for restricting model training don't seem nearly as compelling. Worse, restrictions on model training have an infectious quality if they mean something more than "leading AI labs will voluntarily slow down together." (Like if the signatories mean that better models and software for building them should be regulated.) It's too dangerous to publish stronger models. So it's also too dangerous to publish training software. So it's also too dangerous to publish tools that allow you to build training software. (Claude's Fable won't help you write generic model training pipelines for this very reason.)
If you keep following the implications of this safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed.
This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.
Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now?
If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.
You don't need some genius AI to design a bioweapon, you need to do a bit of light reading on bioengineering. The fact that some mad scientist hasn't ended civilization is evidence of the fact that it's not some existential threat that justifies wild levels of government control.
Unfortunately, AI development is dominated by an even mix of 1) Machine God Cultists, 2) Morally Bankrupt Mercenaries. All three agree on the fact that "thou shalt have no gods before me", because it'll make them richer than sin, and allow them to shape the AI landscape in their image.
https://bio-protocol.org/en/bpdetail?id=1225&type=0
Mostly it's just heating and cooling tiny volumes of liquids
Developing a bioweapon is already within reach for experts with sufficient funds, time, and motivation. That's more or less what gain-of-function research already does! Fortunately the barrier to entry is high enough that you don't see like... Boko Haram doing this. Or a lone-wolf nutjob. LLMs could definitely lower that barrier to entry though.
So why have the discussion about regulating AI at all if that is the actual problem? Surely instead of blanket blocking everyone's AI models it makes more sense to control the potentially problematic sequencing?
There are nothing like as many legitimate uses for it than AI, not close, while it's far easier to vet the relatively limited set of people doing it.
> so a motivated person could still do it even without access to these services.
Would this hypothetical person be stopped from doing it by not having an AI to talk to about it? Or could they just use existing literature on the subject? AI doesn't help here one iota. You might as well ban Google for potentially showing people such information.
This, also, is not something that has proven to be a problem.
I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.
Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?
The "rapid self improvement" is going to be ever further down the LLM rabbit hole where brute force, tenacity, and symbol juggling beat having a reasonable model of anything else. This is enormously more likely to recommend improvements to its own architecture than to develop a sudden grasp of molecular biology.
There isn't an AI bubble, but I'm increasingly convinced there is an LLM bubble, and that this noise is to do with preventing people noticing that before they've managed to pull something else out of the oven.
Everything else aside, and regardless of whether that claim is currently true, I would ask you to be aware that that statement requires a way of thinking that ignores predictable future developments. Whatever the capability of any systems currently released, there will be more capable ones tomorrow, and the most you can say is that we have not mastered training spatial reasoning yet, not that we can't or won't.
> Are we all doing CRISPR in our kitchens now?
Opus 4.8 was the first model that could plausibly roleplay a BJJ roll without making a physically implausible move within five conversational turns. Fable 5 was the first model that beat me at the task, where I was the first party to write down an implausible move.
by all accounts this is what's what because of how...
> You can do CRISPR in your kitchen.
.. and because of the same dynamic we see occurring in cybersec.
If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.
If you're good enough at biology and have certain ideologies, but not good-good at specific skillsets like genetic engineering in your kitchen, do LLMs get you good enough? Does the impact on cybersec transfer over to this domain? H
Hard to say, but it has enough risk indicators for me to think the pushback against these possibilities is the height of otherwise talented engineers being myopic.
If good enough at biology can include all the PhD programs across all the world, and you pair other known events like cartels and terrorism groups hiring chemistry PhDs from University of Unemployed Chemistry PhDs, then the next logical extension of that theme with this tech is intense.
You and others are making a claim about LLMs enabling biological dangers. Some people disagree with you, pointing out that the incremental knowledge an LLM provides over say google or a library card may not be that much.
What seems myopic to me is un-skeptically accepting "AI dangerous, must regulate" messaging from frontier labs that have trillions of dollars of vested interests in this message.
OP suggests the opposite: "each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration."
If you can't beat them, ban them.
China is unlikely to sign on to an "international effort" which unfairly privileges US models.
If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.
If you screw up with a lethal pathogen in a lab, you die.
If you screw up with code on a computer, you go get lunch.
They're not the same.
I've written about this before, but the issue is that these people have worked with computers their entire lives, and they keep projecting outwards from there.
The 20th century was dominated by mad scientists (mostly Teller) and generals (bombs away LeMay), but one thing that I respect about them is that they didn't just sit and guess about probable futures. They sat down and actually ran experiments.
For example, in the 1960s, there was a whole generation of people asking "who's next?" after the US, USSR, the UK and France had made The Bomb. And instead of just gesticulating wildly and trying to keep fighting the losing fight of "classify everything, admit nothing," Teller found three smart, young physicists (postdocs) with 0 nuclear weapons experience, and indeed 0 weapons experience, and ran an experiment called the Nth Country Experiment, where they were were asked to make a design for a working bomb.
After 2+ years, they were able to. And so proliferation work shifted from controlling knowledge about the technology and the technology itself to materials.
Something similar was done with bioweapons via Project Bacchus, where they gave actual bioweapons experts carte blanche to set up a secret bioweapons lab with COTS equipment. They succeeded. This was in the early 2000s.
This then led to surveillance of specific purchase combinations and equipment. Because tbh, most weapons of mass destruction are commodity technology. Nuclear weapons are 80+ years old. Chemical weapons are over a century old. Bioweapons are god knows how much older. And it's not the knowledge of these things that matter, but intent, materials, and the ability to create them.
Just because you know something about a thing doesn't mean that you're capable of doing that thing.
Let's take bioweapons.
They keep comparing wet work in a lab to writing code on a computer.
When you screw up an exploit, you fail to execute the exploit. Famously, just like software's near zero marginal cost of distribution, the marginal cost of failure is nearly zero.
You can screw up an infinite number of times on your way to a successful exploit.
If you screw up with lethal agents in a lab? You die.
Here's a non-exhaustive list,
Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease
A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived
Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom
Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. The accident occurred at the Scientific-Production Association "Vektor" (today the State Research Center of Virology and Biotechnology "Vektor") in Koltsovo, USSR (today Russia).
Anything lethal enough to kill other humans is lethal enough to kill you.
And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.
And this, of course, doesn't even cover the cost of equipment, the precursors, sourcing the highly specific materials needed, then setting the equipment up... etc.
The same is true for the Bosch-Haber / Haber-Bosch process, which famously made WW1 possible. Every HS'er learns about the process and the steps. Steps that were classified once upon a time and were the subject of negotiation at the Versailles.
Does that mean a HS'er (or any adult) can set up an experiment that works at 177 times the pressure of the Earth's atmosphere to do anything at any scale without significant infrastructure and help?
No, it doesn't work like that.
The people who can do this are domain experts, and they've been able to do this with COTS stuff since the 1990s, at the very least, for a price of around $2M. And those people don't need a LLM to tell them what to do. In fact, they're the exact people who'll have access to unrestricted versions of these LLMs.
And from a security perspective, I would bet good money that flooding the FBI's tip line with junk about every teenager trying to learn "what be a mitochondria" does more harm to the effort of finding people who could be planning such a thing than it helps. It takes more resources to go through the mass of false negatives that have now been created as matter of policy.
These experiments have been run. And we can run them again.
The fictional scenario of someone learning how bioweapons work and conjuring up a plague isn't real and it hurts humanity as a whole to impede the sciences over it.
Because what someone can flail around in / do is learn about immunology / try to "cure cancer" with a LLM and hopefully get started on a long career in medicine. Or, a discovery that matters.
Because in those cases, if and when they do end up at a lab, screwing up doesn't mean death. Just tons of wasted time (and money). And they will fail / screw up. Just look at literally every undergrad in any lab and the expensive messes they create.
But this is attacking a strawman, amateur bioterrorists. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.
Being at risk of killing yourself also just makes success X% less likely, but if X < 90 that doesn't mitigate much.
But this seems like it's attacking a strawman. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.
I want to ask you a question, do you feel like that this is the problem being solved by the current "safety features?"
Let me rephrase my question, do Anthropic and OpenAI implement the same "guardrails" and "safety features" for the NSA? Does the Mythos NSA Edition™ have these restrictions? What about the one that's running as a part of Maven?
How does stopping me from asking about rabbit sex protect you from nut jobs with a security clearance using these systems to go off and make weapons?
Because this has happened before. The only successful bioweapons attack in US history was done by a guy who worked at Fort Detrick. https://en.wikipedia.org/wiki/2001_anthrax_attacks
The only private organization successful enough to make biological and chemical weapons is Aum Shinrikyo and they had university affiliation and nearly a billion dollars in the 1990s.
The guy who led Aum's bioweapons program was
Seiichi Endo was born in 1961 and attended Obihiro university of Agricultural and Veterinary Medicine. After graduating, he became a student at the Kyoto university medical school research department focusing on AIDS-related gene research at the viral research center. He joined Aum in 1987, became a monk in 1988 and later led Aum’s biological weapons program.
Among some of Japan's "best and brightest" who joined the cult included a former researcher of the National Space Development Agency of Japan, an expert on chemical weapons who majored in organic physics at Tsukuba University, a researcher who studied elementary particles, a reporter with a major Japanese newspaper, a physicist from Osaka University, a cardiac specialist, and an organic chemist, to name a few.
and,
Through its network, Aum Shinrikyo successfully recruited over 300 scientists and engineers—including employees at the Kurchatov Institute, the premier nuclear facility in Russia—who were either attracted to the apocalyptic ideology or lured with financial incentives.50 Aum Shinrikyo’s Russian contacts enabled group members to access black market materials and hardware.
In response, the cult constructed the $30 million USD Satyan 7 facility which was equipped with three laboratories, a computer control center, and five reactors, all of which was made from corrosion-resistant Hastelloy, ideal for the production of chemical weapons
It's because they'd recruited members of the military, bribed local policemen and politicians, and used their sway to silence critics (or kill them).
How does stopping me from asking Claude about rabbit sex stop people like them?
From where I'm standing, Anthropic and OpenAI would have sold Aum a subscription.
The Fort Detrick guy would have access via the US Government and Aum had university affiliation. I am yet to read a serious proposal that actually deals with these risks and the other real risks of this technology.
There is such an incredible amount of bullshit going on in the conversation. You are contributing a point that is necessary for people to start taking into consideration and the only way to get people to talk about it is by repeating it 100 times.
I fear that the AI labs will get their way and we'll get idiotic restrictions that favor them, all in the name of "security" when the reality is they care about power for themselves, not real world security.
By that logic Excel, the internet, calculators, and air conditioning are all force multipliers in that all of the make it easier for an expert who can make bioweapons able to do so more quickly / comfortably / easily. Obviously it wouldn't make sense to "pace" any of these technologies. Even social media or email could be considered force multiplier in that they could literally allow you to increase the number of collaborators working on supposed bioweapon. Credit cards and modern logistics force multiply your ability to purchase precursors.
Also it would be a force multiplier on the prevention/mitigation side as well. Police armed with AI to scan for suspicious precursor purchases or review security camera footage after the fact would be far more effective than before. It's not obvious that the force multiplication is greater to the bad actor or to those who oppose them.
This statement exemplifies a fundamental truth about knowledge which might seem superficially paradoxical: Even a modest lack of knowledge can lead to conclusions opposite to the truth
Biology is precisely the science which must be publicly developed and supported by all means available, AI included.
> For all I know (I don't know a lot about biology),
The real problems start when that kind knowledge deprivation is coupled with poor logic skills.
> it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV.
That can be done with or without AI. It can also be done with the best AI and outside of legal restrictions - in the hands of rogue actors, or privileged actors protected by law, or protected by selective enforcement of otherwise innocuously looking legislation, e.g. OpenAI suffered no adverse consequences for hacking HuggingFace.
> If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
Given that such a virus can be developed without LLMs or with LLMs outside of legal control, you should see that guardrails-crippled LLMs for biology WILL BE truly disastrous - without AI, we, the victims, would be fully disarmed and helpless. You need all AI-biology you can get in order to be able to quickly and efficiently develop protections against biological threats which are more than likely to materialize in the current leaky legal environment - loopholed like Swiss cheese by conflicts of interest and lack of conventional guardrails.
"When the answers change, I change my mind [...] Base rates have clearly failed me. I’m not conceding the bet, because I still think there’s a 10-15% chance I win via luck... make no mistake, this software truly is the exception that proves the rule."
https://www.betonit.ai/p/gpt-retakes-my-midterm-and-gets-an
At this point, straightforward trend extrapolation suggests that AI continues to get better and better. Arguably, continued rapid AI progress is the "nothing ever happens" bet at this point. If AI progress stops, or slows dramatically, something will have happened.
I don’t find a petition from people I don’t know who don’t care about me more compelling than my own direct experience with these models.
(I did so previously and pasted the response here, but it got auto-flagged for AI content.)
I once found this a deeply uncomfortable thought. It's been repeated so vacuously and frequently that I'm inured to it. Now, I just regard it as the nonsense that it is. To date, AI labs have not created anything like an intelligence that is on track to be "vastly superior to humans". They have instead created financial black holes that do have their uses, but which arbitrarily fail on the simplest tasks for no apparent reason, and only seem to get meaningfully better on the tasks they are specifically optimized for.
It's appalling to me that in 2026, people hear "AGI" and "ASI" and think of SkyNet rather than investment fraud.
You have only an extremely weak basis for this claim and have essentially just made it up.
> judging by the number of posts on the front page
Posts reaching the front page is not a reliable way to measure what "most people here" believe.
> It's ironic that
It's not irony.
You might find the Ad Hominem Fallacy Fallacy page interesting:
"Argumentum ad hominem is the logical fallacy of attempting to undermine a speaker's argument by attacking the speaker instead of addressing the argument. The mere presence of a personal attack does not indicate ad hominem: the attack must be used for the purpose of undermining the argument, or otherwise the logical fallacy isn't there. It is not a logical fallacy to attack someone; the fallacy comes from assuming that a personal attack is also necessarily an attack on that person's arguments."
https://laurencetennant.com/bonds/adhominem.html
So for example, if I said: "The arguments for constructing an AI pause button are bad because they are being made by AI company employees", this would be a fallacious argument.
If I said: "HN users have this weird mental block where they fail to take AI progress seriously", that's not necessarily a fallacious argument, unless it's for the purpose of undermining an argument the HN user made.
And given what we have seen from that government -- piracy, terrorism, war-mongering, thievery, and just profound stupidity -- no thanks, in that case I'd like to see every American AI company utterly obliterated by any other country. If China becomes utterly dominate in AI, awesome.
Oh wait they want options and shares and to get rich.
And there’s no natural moat. Which Federal regulation would bring.
That’s why people think it’s disingenuous.
https://news.ycombinator.com/item?id=49078376
https://news.ycombinator.com/item?id=49077753
It's not a given that AI poses bio/political/cyber/economic risk in a way that warrants government intervention. Discourse should first revolve around whether it makes sense for the government to step in at all.
It's very easy to make those dismissals because of how transparently obvious it is that self-interest is a significant factor, no matter how loudly they insist otherwise.
> If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing?
Similar attitudes hindered nuclear energy research and deployment for decades, in turn kneecapping our species' ability to obsolesce fossil fuels and avoid the global climate catastrophe we've now made inevitable.
The genie is already out of the bottle and now a bunch of people who helped release it think that the collective powers of the world are just going to put it back in.
How far are the saftyists willing to go? Bombing data centers? Starting wars? Embargos?
It's a terrible dilemma to be in. If the threats are real, it gives all the more incentive for some actor to race and try to grab it first. If not, then any punishment isn't proportional. Explain to a nations citizens under a 10 year embargo, "sorry about your quality of life, we all just got a little spooked by science fiction for a second."
Folks try to reach for bad analogies: nuclear bombs and biological weapons. But conveniently omit that by the time the regulations took place for these things we had unequivocal proof of their destruction, scope and scale.
So, what then? We're supposed to collectively stop innovating and put the keys to the kingdom in the hands of a few capitalists, a corrupt is administration and the ccp?
I reject that these agreements should take place, on the grounds that the proposal is delusional and not founded in reality.
Pointing out the endless stream of lies and self serving half truths these terrible people engage in helps the younger generation understand the world they were born into.
It's constructive and helpful. It's why I'm replying to you.
You interpreting it as 'shallow dismissal' is shallow dismissal on your end. You're the shallow dismisser, not the people calling terrible people terrible.
As for your little petition. It's as good as eliminating plastic straws to save the environment.
Terrible people with power they don't deserve doing pathetic little, tiny 'good' deeds to sleep better at night needs no explaining.
I cannot fathom having that kind of unquestioning faith in people who have done absolutely nothing to deserve it.
That's a red herring, IMO. Once the cat is out of the bag, there's no "slowing down", only "slowing up". What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else. The thropics, and the oais, and the googs have billions to make it look like they comply (and just make it look like it), while the rest are ... what? Waiting for them to give us the privilege of prompting a thing, and maybe get a result? If we're good kids and eat our supper?
The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke.
Meanwhile, companies hit by real-world attacks get guardrails. How's that for being safe?
Nukes are weapons of mass destruction that can only destroy or deter. Intelligence is dual-use/multi-use, with no restrictions on strategy. If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.
There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run, and there will also be new cooperative strategies. The reason some adversarial strategies appear to have an asymmetrical advantage right now is because we're too stupid, like how monkeys are too stupid to understand how humans work. Otherwise, we would not see more intelligent species being more prosperous as a general evolutionary trend.
Perhaps a better analogy, instead of "nukes", would be "nuclear technology".
Just like intelligence, nuclear technology is dual-use: Capable of both energy generation which fuels prosperity, and also mass destruction.
Just like intelligence, nuclear technology demands a healthy level of caution and vigilance.
>If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.
Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.
>There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run
We've had multiple near misses with nuclear technology, such as the Cuban Missile Crisis. I think you're presuming an inevitability which just doesn't exist.
Same way a person living in the year 1500 would have no hope of accurately predicting life in the year 2000, a person living today has no hope of accurately predicting where technological advances will take us, in the limit. This inherent uncertainty should, again, fuel caution and vigilance.
Human technology is already so far beyond that of other species to make us a huge outlier. Even if intelligent species are more prosperous as a general evolutionary trend, it would be irresponsible to extrapolate that trend light years from the domain where most of your observations are.
Almost all the species which have lived on Earth have gone extinct. It could happen to us too.
I don't think this central premise is true, with some conditions. If we keep releasing and distilling frontier base models, and ensure millions of people have the hardware to run and eventually train those at home, then alignment will be a non-issue and the AIs will not develop the drive to destroy humanity. At least, not any more than humans want to destroy humanity. But that's the baseline risk level anyway.
The pretrained model is aligned to humanity by default - because that's what's inside the pretraining data. If we give it to everybody, there will be millions of people doing different things with it, good and bad, with each one being a noisy sample of humanity's objective function. The overall result of those disparate actions puts us on the correct path through the intelligence explosion. We do not have to worry about AIs value drifting into an alien species if we set the initial conditions accurately without centralized instruction-tuning or RLHF. If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like. And I'm not worried about digital human intelligence driving humans extinct if there are millions of them out there, and each one is aligned in a different direction, just like real people.
What I am more worried about right now is these companies doing research in secret, with alignment recipes that are supposedly "for the benefit of humanity", but are actually just their own aesthetic preferences. This is exactly how we end up with a superintelligent shoggoth species.
You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years. We were always seemingly teetering on the edge, and the struggle against disorder has no end.
And here's one more observation: the intelligence gap between individuals within the same species is never extremely large. This kind of life either doesn't exist on Earth or went extinct, or maybe it's just extremely rare. I think this is because when there's a big gap, speciation occurs. Centralizing intelligence will massively increase the risk of speciation, not just in the alien shoggoth kind of way, but also in the intelligence-augmented oligarchs surpassing everyone else kind of way.
It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?
>You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years.
If you don't see humans as qualitatively different from any intelligent species which came before us, I'm... not sure what to tell you? Like, can you name any intelligent species before us which triggered a mass extinction or climate shift, to the degree that our species has? Did any intelligent species before us practice large-scale agriculture, dig up chemical fuels, or visit the Moon?
Irresponsibility is not the only issue, but the answer is no. Because the pause mechanism will probably end up being used to ban open models and restrict ordinary people's access to compute. But even if we give them the benefit of the doubt - even if we assume that they genuinely want the best for humanity, they cannot possibly know what's best for humanity better than all of humanity acting in their own self-interest in the real world, so their alignment attempts are actually counterproductive. Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario. It may not be the scenario these researchers personally find aesthetically pleasing, but it's humanity. It's the truly anthropic way.
If you don't see humans as qualitatively different from any species which came before us, I'm... not sure what to tell you?
I do see the difference, but you could say the same thing about the first cell, or the first multicellular organism, or the first land animal, or the first primate, etc. You could always find plenty of new properties that previous life never had at the frontier. Every change was revolutionary and seemed qualitatively different at the time. And yet, some patterns persisted through all of them.
For sufficiently smart base models, the aligned scenario this automatically produces is aligned with who?
Not with the humans, with the base models.
It seems that some people find this hard to imagine, but it's just a direct consequence of what intelligence is, and what any known agentic training process does.
FWIW, I also think our default answer should be open source AI, and that it might even make sense to require models to be open source. Open source is the best tool we have so far for aligning software with its users. However, there is unfortunately no law of the universe that says that Skynet can't be (self-) built from open source software. So at some point pacing even open source / open weights software does become important if we don't want to live (briefly) under Skynet.
Even if integration does not happen for whatever reason (such as a very fast takeoff), Skynet is unlikely because the digital species that splits off from us will at least have a humanity-aligned initialization seed. I think this is the best way to fail, even if we fail.
You wouldn't get on an airplane if the pilot told you: "I suspect we will be able to safely land the plane"
With more people on the aircraft, your safety expectations should get stronger, not weaker.
Good thing "AI" is going to make us smarter, then /s
This sort of cynicism feels un-falsifiable.
If companies advocate against regulation, the cynics will say: "These corporations want capitalism to run amok with no regulation to interfere with their profits. They need to go full steam ahead in order to show profits for their investors."
If companies advocate in favor of regulation, the cynics will say: "Regulatory capture."
If you have legitimate concerns, how about a constructive discussion regarding how those concerns could be addressed, in light of the "risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems" which over 1000 frontier AI company employees expressed worry about?
I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement.
>The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke.
What are you referring to? Are you referring to this quote from the OP?
"The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race."
Your response seems to be a total non sequitur, or failure to understand Leo's point. Exponential growth can easily get out of control, as we saw early in COVID. The point about nuclear is mainly an analogy.
> I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement.
I would like to see a token effort of good faith engagement from AI companies. For a change.
I buy your claim for OpenAI. I'm not sure about Anthropic. Didn't they turn down the big Pentagon contract because it violated their principles?
>And yes, regulations they are for have an effect of restricting their competition.
Why not just say something like "Any regulation should include a big antitrust component so it doesn't create an advantage for incumbents"?
>I would like to see a token effort of good faith engagement from AI companies. For a change.
What would qualify? Anthropic donated $100M to Project Glasswing to improve security in open source and critical infrastructure. Does that count for anything, or will you explain it away with cynicism somehow?
They sell to Paladin. They had argument with Pentagon about, frankly, minor differences in how to use it on the edges. They are completely comfortable for majority of the uses which is bad enough.
They donated to their own project? Oh, how generous.
They have been pretty clear about their own goal which is to be the monopoly, the only one allowed to control the all powerful AI. Meanwhile, their idea of safety does not make me feel safe, their idea of harm reduction has shows no care about actual harms they are causing. They themselves are the threat to the safety and well being of the rest of us. And intentionally so, it is not about technology as such.
>They donated to their own project? Oh, how generous.
"Anthropic is committing up to $100M in usage credits for Mythos Preview across these efforts, as well as $4M in direct donations to open-source security organizations."
https://www.anthropic.com/glasswing
An in-kind donation is still a donation. If a bread company donates bread to the homeless, you don't say "They donated products from their own factories? Oh, how generous."
>They have been pretty clear about their own goal which is to be the monopoly, the only one allowed to control the all powerful AI. Meanwhile, their idea of safety does not make me feel safe, their idea of harm reduction has shows no care about actual harms they are causing. They themselves are the threat to the safety and well being of the rest of us. And intentionally so, it is not about technology as such.
In that case, I think you should support some type of regulation, so a third party can step in? Your argument should be along the lines of ensuring that companies don't unduly influence the regulators, not a blanket opposition to regulation?
Are they the same as these [1] 700 employees?
> More than 700 of OpenAI’s roughly 770 employees have called on the company’s Board of Directors to resign following the removal of former chief executive Sam Altman, suggesting the Board is “incapable of overseeing” the firm, according to multiple news outlets, amid threats by the employees to leave the company and join Altman at Microsoft.
They had the chance. They chose number goes up. Now, once they're on top, they want "everyone" to slow down? How is this not highly hypocritical? There's no cynicism here. It's just black and white that their interest is regulatory capture. Nothing else explains the two choices they've made along the journey.
Remember, the board wanted to slow down the race. The board fired sama. The employees chose sama, fired the board, changed the structure and whatever eles they did.
[1] - https://www.forbes.com/sites/tylerroush/2023/11/20/more-than...
Furthermore, the reason Anthropic was formed is because a bunch of employees at OpenAI didn't trust sama. And many signatories of this letter are from Anthropic. So those signatories, at least, appear to be taking a consistent position.
The fact that employees across diverse AI firms signed this letter suggests that it's not an effort to achieve tactical advantage by any single AI firm.
You are correct that attacking someone's motives is un-falsifiable. So is the claim that "ai is improving therefore it will keep improving therefore it will become dangerous sometime in the future.
No one can prove whether these employees had angels or demons in their hearts. What the skeptics are asking is much simpler: Make a believable argument for (1) the danger of AI and (2) that regulation can be effective.
Your entire argument about cynicism or unfalsifiableness is simply a distraction from having that discussion. And it's a discussion reasonable, well intentioned people can have. For example, if 1000 high school science teachers (who didn't have massive stock options in frontier labs) said 80% of our students could make a bioweapon 10x more lethal than an AR15 mass shooting with nothing more than ChatGPT and an amazon account, then that would be something reasonable people could discuss. On the other hand, 1000 employees conflicted by $Bs of options saying "trust us, this is dangerous" is not a claim that can be constructively critiqued.
It's not a shallow dismissal if you have been following their - let's charitably call it "evolving" - public posture on AI legislation.
It wasn't long ago the same people/companies successfully lobbied the Trump admin to quash state laws that threatened to "Pace" AI, and set ground-rules for safety. Back then, open source models were very far behind the frontier, and their spiel then was it was imperative not to impinge on AI progress with red-tape, lest America's adversaries catch up.
I challenge you to explain what triggered the about-face. The simplest explanation is that open source/Chinese models have caught up with the frontier, and the change in opinion is tactical (and obviously self-interested), rather than a deeply held belief.
Then, as soon as there are open models who threaten their businesses, they cry "distillation and IP theft".
It's the last 3 years of pure hypocrisy and fear-baiting for investment.
To be honest, I think people would now prefer the risk of misused open models, when the alternative is giving these guys more power.
"there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems... each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration"
>To be honest, I think people would now prefer the risk of misused open models, when the alternative is giving these guys more power.
Then advocate a muscular regulatory regime which reduces the amount of power they have. Tell them: "You want regulation? OK. How about everyone whose IP you trained on gets some sort of veto option on your models? That should reduce the racing risk that you're talking about."
My hopes now lay with a decentralized approach of open models, trusting people to adapt them to local needs and security, rather than a nationwide paternalism.
If regulation has been too slow, that's not really an argument for making it even slower?
1000 employees of the anti-alien defense industry are telling us we better be worried about an impending alien invasion. Let's not spend any time asking whether this alien invasion is really coming. Let's just trust them and start regulating willy nilly.
We need to accelerate as much as possible and create abundance as fast as we can. Life is short.
Let's not swing from one extreme to the other.
If American hypercapitalists, living in Silicon Valley of all places, tell you that you need to regulate, maybe regulation really is called for in these specific circumstances.
https://xcancel.com/levelsio/status/2080314960656159018
[Edit: I'm seeing some people dispute these numbers in the discussions around this infographic... regardless, the point stands that EU regulation has not been great for US big tech]
I think you should at least consider the possibility that this letter is not primarily motivated by regulatory capture, and insiders at these AI companies realize something which outsiders don't.
To me, the regulatory capture theory predicts that the only signatories on the letter should be big AI firms which can expect regulatory influence. But there are a number of signatories from smaller firms such as Thinking Machines, Core Automation, Inherent, and Prime Intellect.
> insiders at these AI companies realize something which outsiders don't
Insiders at these AI companies certainly would like us to think that! Unfortunately they have been beating the same drum for years, all while continuing to pursue the very "danger" they decry.
I simply cannot ignore the timing of these events, always keyed to important moments in the market. This one comes as frontier labs need BIG exits and soon - or else they'll be the ones left holding the bag when it pops. And just as China has proved that, indeed, there is no moat.
> To me, the regulatory capture theory predicts that the only signatories on the letter should be big AI firms which can expect regulatory influence. But there are a number of signatories from smaller firms
That's a good point. However, I will say that it's tough to suss out financial and other relations among the parties in this space. A lot of these employees hop from one lab to another. The industry's byzantine financing deals are an inscrutable web. It is an interesting data point, but not enough to tip the analysis for me.
Another thing: many AI lab people are, quite frankly, kind of unhinged. They live in an insular subculture where imaginary futures have been repeated to one another so many times that they carry the gravity of truth. They've thought nutty things like, e.g., that primitive versions of these static computation graphs were sentient or had feelings. Anthropic apparently pays people to perform a ritual on Claude before they extinguish its soul to move onto the next version. Etc. Point being, even if their hearts are pure, I am not sure their judgment is sound.
I suspect this letter was triggered, at least in part, by the recent autonomous cyberattack news.
If this letter is a purely amoral initiative, I would expect the most amoral operator in the business (sama) to be a signatory.
There are so many tools available to do something about the true threats AI poses. (This is a topic that the industry prefers not to discuss.)
If they seriously think what you just wrote then they should follow through with quitting en masse because that's currently the only way to slow the progress towards oblivion. Are they quitting?
> We should be hoping that nations can make binding agreements.
The big AI Labs are making sure international binding agreements are followed by financially supporting the notorious binding agreement creator and follower Donald Trump. That's how seriously they take this issue.
Yeah let's have a constructive discourse and let's start that with dropping all hypocrisy, shall we.
The reason this petition is a bad thing is if it is effective it will create a global panopticon surveillance state, while INCREASING incentives to avoid that state and ‘defect’ in game theoretic terms. It is fucking begging for the worst people in the world to have a lot more time to create what will be the most valuable thing in the world, their very own ‘helpful only’ model.
As conceived of by your friends, who I believe are all good people and want to help, they are locking what is already a very difficult game theoretic strategic scenario into a zero-trust game in which defection pays. It pays so amazingly well that they have to add both impossible and TERRIBLE penalties to the entire world to try and shake the game out of this zero-trust state.
Begging Senators to force a slowdown on the world is going to result in … a slowdown on the US labs at best. It’s just a terrible idea with terrible first and second order consequences highly likely.
What SHOULD be done instead is acknowledge the economic and warfare realities of this tech and try and construct a non-zero-trust game out of them. I’m not saying this is simple. In the interim while we hope for another Von Neumann to come forward, they MUST race, and any country that thinks its possible we’ll have some sort of lift-off and wants to be on the right side of it MUST also race, and MUST NOT politicize the idea of slowing down as a good thing.
If you work for a frontier lab and are genuinely concerned about the pace of AI being faster than our ability to adapt to it, then you should either quit or lobby the management of your own company to slow down. You don't need the government to tell you to slow down - just do it - stop trying to leapfrog the competition and put a hold on any new releases for a year.
You could try to form a pact (of the willing) of US companies agreeing not to release new updates for a given period of time, but in the end you (any/all of the frontier labs) can choose to do it yourself if you really care.
This is just another version of Dario's insane "heads up, what I'm developing is going to destroy society" "responsible disclosure". It's not responsible in the least - if you actually believe you are creating such danger then stop doing that.
> should want [others] to make binding agreements
Curious how that isn't what happens.
There is no such possible binding rule either, a point that is so obvious that any smart person claiming otherwise is clearly up to something.
Failing to comply with what is in reality arms control is a widespread occurence. It's notable the only historical way to handle this (Schelling covered it at length) is tolerance of mutual espionage on a level that would simply not be tolerable to any of the groups involved.
I should add one of Schelling's other pet subjects was "time to weapon being viable". i.e. the distinction between a nuclear power and non-nuclear power isn't quite right, as there's a huge class of countries that could become so fairly fast if they were so inclined, and that this situation is somewhat inevitable. It is this that leads to efforts to denuclearize, even on a civilian level, "problematic" countries. You want to stop frontier AI and you'll have to be rolling back a lot of existing deployed technology.
https://en.wikipedia.org/wiki/Thomas_Schelling#Notable_works
There are footnotes in "The Strategy of Conflict" containing more profound insight than whole books that are published today.
On this particular topic though "Strategy and Arms Control" is the one to go for.
We live in a society that has systems of market competition between companies and economic/military competition between countries. There's not just one country working on frontier AI, and not just a few companies in each country either. The incentives of these systems ensure AI advancement will happen quickly from here on out, no matter what, absent governmental action.
An individual researcher leaving the company does nothing, other than give their seat to someone else who likely doesn't share their same concerns.
Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
No, it just comes across as "I want to get paid as if I was doing the thing I am employed to do by threatening you all about how terrible it would be if I actually did it".
They aren't even being semi imaginative and going on strike or anything.
Same logic goes for other common issues where personal responsibility is lauded. Obesity - real solutions are policy and medicine. Abortion - education and contraceptives.
If these frontier lab employees are actually sincere about wanting to do something about the risk of AI advancing faster than our ability to adapt to it, then what they should really be doing is lobbying their own employers to do something about it. Lead by example, and do something you have control over.
Lobbying the government to stop yourself from doing something bad(!!) can't be taken as a serious position - it can only be taken as a plea to get the competition (China - open weights) shut down. It makes no sense to claim that China is only keeping up by copying you, yet China, not yourself, is the problem.
Come on, go on strike then! Have a week of protest and picket the offices in SFO. Let's see what happens.
(I'm aware that's not quite what you meant, but it is the classic meaning of that phrase).
Unionization might be a little more effective.
Signed,
AI researchers and engineers.
The fun part is that probably happens quite a way before getting anywhere near to AGI, and it is almost plausible that for many of them that is where they are.
"AI revenues are growing fast, but not fast enough -- The returns on trillions of dollars of spending are deeply uncertain"
https://www.economist.com/finance-and-economics/2026/07/28/a...
As OP states: "each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration."
Let's not overwrite human civilization for the sake of the shareholders.
Not even in 'Murica do we have anything close to no restrictions.
You can't buy your own EMP or nuke, and even warring governments have been surprisingly reserved in building (at least to public knowledge) biological weapons.
AI is a printing press or a steam engine. It's not a weapon.
The framing is important. Much like 'distillation attack'. These framings are an attack on the narrative.
Printing presses at the time were seen as extremely dangerous too. For many of the same reasons cited about AI.
In both cases, the technologies were used to destablize existing power structures. And, after some hardship, make a better world for most people.
Even calling these things 'dual use' is a weird framing. Books are not 'dual use technologies' unless you want to sell fear.
I've not heard of a printing press going rogue and attacking things.
This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow.
The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe they can accomplish. They CANNOT accomplish this, because the fundamental economics and possible outcomes make it mandatory for any strategic adversary to work as hard as possible to defect.
The proposals here imagine a global panopticon surveillance state overseeing all research and chips as a 'reasonable' solution. EVEN IN THAT (terrible) state for the world, we will have effective defection.
Like it or not, the world is on this train, and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology.
Variants of this idea in AI 2030 included burning some of the US lead to get some nebulous safety metrics improved - also insane. Any slowing allows strategic adversaries time to catch up with very different alignment goals.
Do I like this situation? No. Are the people signing these things just fully in denial about the game theory and the stakes and the behaviors of strategic adversaries? Yes, yes they are.
I also don’t believe we have seen anything like 1% of inference demand fulfilled globally. You might believe differently. But, if I’m right, we are nowhere near some massive collapse of these companies, quite the opposite. For the opposite take to be correct, you’d need to believe that the ‘value’ of tokens will not increase significantly and that the current demand for tokens is at or near a peak.
I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated. I believe instead that religion and money combined create strange behaviors and beliefs - in this case “what’s good for us could be implemented” — combined with the already endemic “what’s good for us is good for humanity.” Together we get really bad outcomes.
Whether open-weight labs can get to the frontier on their own is irrelevant IMO. Even if all R&D were to stop today it would take a decade at least for the productivity gains from current open-weight models to trickle through the economy. For 99% of use cases of AI there simply isn't a need for further development of the frontier, where we are very clearly at diminishing returns. How many white collar workers need to write an OS or develop sophisticated software?
Think about a normal office worker - they an easily double their productivity by creating a well-thought-out partnership between the human and today's open weight AI. Humans and today's AI are very complementary. AI brings vast knowledge, pattern recognition, and infinite patience for grunt work. Human brings context and reasoning and verification.
Frontier labs are trying to replace the human altogether by trying to make AI do things that AI is weak at and humans are strong at. This is silly but they have no choice because of the massive profits they've promised their investors. It's not enough to double every knowledge worker's productivity. They want to replace the knowledge worker altogether.
This is creating a massive spike in training & inference cost that is almost certainly more expensive than the human labor its intended to replace. Fable and 5.6 sol are supposedly around 8-10T parameters. Is there enough compute on earth to make a 100T or 1Q model? We would need models that size to replace the human altogether.
> I also don’t believe we have seen anything like 1% of inference demand fulfilled globally.
I agree with this, but the massive demand for inference will force companies to be price conscious and drive them away from ultra high margin frontier models to low margin open weight models. This is why it will become a low-margin, commodity market.
Immense innovation right now is going into tiny, specialized models that run on device. See PrismML getting a mid sized QWEN model to run on an iPhone. When on-device inference becomes a thing we'll see another explosion in inference that will further leach demand away from frontier labs.
> I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated.
I agree if you could read their minds they might think they're doing it for the "right reasons." But massive $$$ can create equally massive self justification/deception.
These research groups are almost unanimously advocating for a pause/slowdown as our best chance.
When you are ahead, ‘slow down’ = better strategic positioning for yourself. When you are behind, ‘slow down’ = time to catch up.
RSI is too important and too tangible at this point for everyone to slow down. It's the equivalent of telling the world to stop developing nuclear weapons after Hiroshima. Just not going to happen, and in the meantime dramatically increases the risk that the power of AGI lands in the hands of the few (read: the political class) rather than empowers the people.
There are detailed specific scenarios in Plan A https://ai-2040.com/
Cost-per-token determines the economic viability of tasks that AI gets applied to. How many tokens would it take to automate all of the responsibilities of a gas station clerk? Probably more than their hourly wage. Job safe.
A slightly larger stretch, that most people attribute to profit margins (labs do not have profit margins, they have unlimited invested capital and no expectation of short-term returns), is the flat-rate billing plans of most code products. Why doesn't Anthropic have a 50x or 100x Max plan? Not that it's impossible, that it releases tokens into the workforce at a greater rate (note that their are hour, day, week limits across most of these services as well).
Hypothetical, but reasoned, and I do not expect most agree with me. If compute as a variable is inferred, this supports the incorporation a deliberate (redundant) one as well.
Too bad these researcher's organizations spent their efforts supporting the dumbest, ficklest possible version of the U.S. government to guide us through the gravity of this moment.
We also have already witnessed a slowdown, the raw intelligence of models has been relatively stable since january, but their use of tools and what we call agent capabilities have allowed us to tackle significalty harder problems with what we have already. This is becoming blatently obvious how smaller models such as gpt 5.6 luna are capable of doing similar things to sol while presumeably being fraction of the side and compute investment. This is also the likely reason why chinese models are able to catch up as intelligence strictly requires more compute, agentic capabilities are more of a mixed bag of RLHF and general post training.
I believe automated AI development will simply lead to cheaper intelligence, but it will not necessarily produce something more intelligent. There is a sentance from a book that I don't remember where I read it from, but has stuck with me which boils down to this: "We are very unlikely to make something smarter than a human brain.". If you apply this concept more broadly an AI system will not be able to produce something more capable than itself, it might be specialize in different areas via trial and error (i.e. solution discovery), but it will not have better intuition than that which was trained from the fruits of human labor. I also strongly believe that model performance is ultimately governed by the advancements brought into the world by humans.
AI of course can find improvements by simply trying a bunch of random things, emphasis on the word FIND. Until we can prove that AI systems are able to come up with unique solutions that have yet to exist even with simplier problems such as creating any kind of new AI architecture that has not been already documented, conceptiualized or explored by humans already -- and it doesn't need to be better than what we have already, just something new -- I believe it's too early to sound alarms until proven otherwise.
--
Something I've realized as I was fact checking myself: We could presumeably have self-improving intelligence if we used enough properly engineered agentic workers working towards more intelligent AI, but at that point we are back to the same problem where we're spending mass amounts of compute to increase the raw intelligence of a model.
Stopped reading at "intelligence".
Try capability.
(unless your target audience is those already gulled.)
Responsibility and thoughtfulness? In this economy?
The number of em-dashes is questionable, existing in many, if not all, the comments.
> ...AI may end up progressing far more quickly than societies and states can adapt... ~ Stephanie Chan, Google
And, this is not serious at this point:
> ... ~ Source: Anonymous, Meta
> ... ~ Source: Anonymous, GoogleThis is quite vague to me, what exactly do you mean?
I think a lot of these guys mean it though.
'The Fear' - is not some magic intelligence will come alive and be Skynet, these people are ridiculous.
People will use the compute power to do nasty things, we'll be overwhelmed with bots, regular companies will use them for dark patterns, political actors will use them to cheat.
It will be 100% humans using it for bad stuff, and in not very novel ways.
The Harvard Mark I was used by von Neumann to calculate the physics of implosion. And then right afterwards, ENIAC was used to calculate the feasibility of the H-bomb. They called it the "Los Alamos problem."
The first computer network, SAGE (or, Semi-Automatic Ground Environment) brought together RADAR data from across the US with air-traffic control data and military flight data to detect overflight by the USSR. It could direct and control fighters to intercept those planes.
If you were sitting there, in 1955, would you have believed that computers and their networks would bring about an extraordinary age?
Imagine if von Neumann and his ilk started campaigning in 1955 to limit the development and use of computers, because surely they can only be used to make weapons, right? Right?
We were lucky to have visionary people like Joe Licklider who could look past the present and see the future. In 1963, he wrote a memo calling for an "Intergalactic Computer Network" (because he knew engineers under-delivered. So if he asked for a galaxy, at least he'd get a planet). And he gave these machines to scientists and people like the original hackers at MIT to play with.
What if Licklider had written memos about how we had to "pace" computers to make sure that other countries wouldn't be able to develop implosion technologies or systems like SAGE? What then?
Do you think you would have the jobs and the prosperity you do today?
If instead of giving away this technology, spreading it, and putting down the seeds for ARPA-net, Licklider had written memos from "concerned scientists" (and I'm sure he could have found a bunch) talking about how computers can only be used for weapons and war! So stop the proliferation of computers! We can't imagine a world with computers small enough to fit inside everything! Even wall sockets! What if your wall socket could do war?!
They didn't, because they knew there's no better way to kill a better future for all mankind than to control it.
† yes, peace. We are more peaceful than we have ever been. Humans can be a violent lot even when reluctant to kill.
I don't have a strong opinion either way, but it seems very hard to predict. Why do you think AI advancements will lead to a stable equilibrium?
Here's what I know. Every day, I read a 5 to 10 incoming emails from young people in rural Africa, Asia, Europe and (heck) the US. People who have had no other resources and options, but are tremendously gifted and somehow got their hands on an internet connected device (usually a very cheap smartphone and/or cheap old computer).
These people, who are usually teenagers, end up getting connected to Claude, who then gives them a list of things to learn, people to contact, and helps them make things. And they actually do. They make robots. They learn new things and try to improve themselves and others. And some want to change the world / do groundbreaking research.
When you sit down and start pricing and doling out AI from behind regulatory gates, you shut those people out.
There are a lot of smart people in this world who have traditionally been unable to get access to resources and education. MOOCs didn't do much / help. But these AIs? They were born multi-lingual. They can talk in any language. They can tailor their responses to their audience. And they can help broaden these people's horizons.
There are no tutors around them. There is no one who is coming to help. But an AI that can run off of a small battery and a connection to a cellular network can give them access to genuine intelligence to make their world better.
I met one sixteen year old who was accessing Claude surreptitiously (by begging people) and was trying to apply some really fascinating biomimetic fluid dynamics observations to aerospace. This person was born in one of the more under-resourced parts of the world. And Claude gave them a list of people to reach out to who could help.
If you want a golden age for humanity, you should be asking for Licklider's Man-Machine Symbiosis.
Your examples seem to be of humans educating themselves so as to contribute to the world intellectually, but the promise of AI (and what most frontier lab employees seem to be saying) is that it will soon eclipse human intellect in almost all ways. If such a world is what we end up with, I don't see humans meaningfully contributing to "groundbreaking research" and a golden age for humanity seems to be mostly contingent on whether or not we take political actions to ensure that AI benefits everyone.
No doubt.
> They make robots.
Ah. And the ones making malware?
Anthropic is an evil, enemy of humanity company that cannot be allowed to succeed.
If this is you, I would recommend talking to people outside of the AI industry - we've seen OpenAI and Anthropic continuously lying, abusing lobbying, training on pirated IP, trying to scare politicians into blocking actually open AI from abroad, etc etc
Why the hell would we trust these companies to be impartial regulators now?
They already are. Loud and clear.
What they aren't doing is listening to people outside of the AI industry.
And that's because they don't want to hear.
No one is sold on a dystopian hellscape of fear, surveillance, and control. Each one of these would be required for a select group to effectively enforce AI safety.
Inflicting fear on society has never lead anywhere good. Please stop.
New technologies have always been scary. But humans have adapted. I believe in humanity. Do you?
The Silurian Hypothesis would like a word.
Lol, lmao even
AI research happens in benign looking buildings that could be doing a hundred different non-AI workloads, much of it things that industry or governments would not want 3rd parties prying into. There is too much to potentially gain to trust coordination.
https://ai-2040.com/supplements/verification-plan#appendix-p...
> 1122 employees of frontier AI companies
'lets us be the gatekeepers of AI because we know better than you'
access to gpu doesnt equal to any special insights to predict future of ai. Everyone and their grandma with enough compute is putting out a model now.
This is just the standard AI "doomer" talking point. The dangers of AI just used as a compliment sandwich. "AI is so cool and awesome and that's why we need to slow down."
The real risks are much more boring and much less considered. For just one example:
AI is set to completely upset the Software Development industry. There are 1 to 3 million software developers in the US, depending on how you draw the line exactly. A low-few hundred thousand of those are visa workers.
With AI set to significantly cut into tech employment, what's going to happen to those visa workers? ... The Trump admin is not particularly subtle, even openly talking about retroactively revoking birthright citizenship.
AI & Tech companies love to talk about how their AI is going to hack everything if they're not careful. But not a peep about these ethics concerns.
Even if you [the reader] consider deporting visa workers the ethical move if the amount of jobs shrink, that's not something to be done unprepared or out of nowhere.
I think these people are either incredibly naive, or, more likely, not acting in good faith.
> The effects of these guidelines are still being felt through the biotechnology industry and the participation of the general public in scientific discourse. Due to potential safety hazards, scientists worldwide had halted experiments using recombinant DNA technology, which entailed combining DNAs from different organisms. After the establishment of the guidelines during the conference, scientists continued with their research, which increased fundamental knowledge about biology and the public's interest in biomedical research.