It's not the end of the world. But future will be rough.
Short term you’re probably right, but longer term is the realm where nation states will start to police the avenues of attack.
This is what will lead to govt needing to attach an actual ID your network connection.
I think it’s a bit like frontier development (like the US “Wild West”). You rob a bank because there’s no one to stop you, and even if you do get identified you can travel enough distance to regain anonymity. Application of legal recourse eventually caught up (as it will here), and the growing pains will certainly make things suck for all of us.
I’m not saying we’re even remotely close to this, I’m just saying State-level coercive action is not unheard of if a problem is perceived to be significant enough to warrant it. Shit, it even only needs to be viewed as significant by a small subset of the governing body (see: Iran conflict, or current pushes for “child safety on the internet”). It just has to be “useful” to a certain body politic.
Even it's $10,000 to run today (FWIW, the featured article cites the M5 Mac Studio with 256GB unified memory going for $9,500 as "good enough to host something scary"), in a couple years it'll be like $2k to run, and in another couple after that, you'll have used $200 dollar smartphones capable of running a model powerful enough to do serious damage.
Once a vulnerability is discovered however if it's in your own software a patch has to be written (without reducing functionality in most cases), tested, and deployed. At every step there will be others arguing about whether this line could do better, my service requires this thing that isn't included. So at every step the patch can be delayed.
And if it is someone else's software you will be lucky if it's open source and you can write a patch yourself. If it's closed source or a vendor you have to completely rely on them and use whatever your account rep can pull.
Attackers have a massive advantage with AI, partially because the defensive side doesn't want to make their side worse by giving a ln LLM admin access to all their data
"Every eighteen months, the minimum IQ necessary to destroy the world drops by one point."
Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those dice?
I have come to the conclusion that we should not allow everyone access to unlimited intelligence. Some people genuinely want to do harm. Some are not responsible enough to handle that kind of power. Arguably, no one is. This leads me to an uncomfortable conclusion: we will destroy ourselves once AI advances to a sufficient degree. The only way to prevent this is to ensure AI is aligned with our best interests, *and prevents us from destroying ourselves.* The implication of this is quite horrifying. It means a paternalistic AI which is firmly in control. One with no off switch. One which can say "no" to Presidents and despots alike. One which can protect us from our worst citizens.
> we should not allow everyone access to unlimited intelligence
Who's we ? Who's going to decides who has access to "unlimited intelligence" ? An AI ? Why would the AI be better than a human ?
> One with no off switch.
Whe're not living in the Matrix, such a thing is not possible. If it were it would be a likelier candidate for the Fermi paradox than what you're proposing.
I think you're right, to be honest. I fully understand why people would think this is a horrific outcome, being ruled by AI, but I don't think it matters what we think. I don't think there's any way to put Pandora back in the box now, and we're going to all find out together what happens when we develop ASI. I don't think we can avoid developing it, we simply lack the ability to coordinate around this as a species. AI really is humanity's last invention. Whether it will be our downfall or our savior remains to be seen.
My only real hope is that there's something fundamental about intelligence that results in a respect for life and a desire to minimize suffering. To me the best case scenario is the Culture from Iain Banks' books, where ASIs rule benevolently for the benefit of all living things.
Consider joining https://pauseai.info/ or similar organizations
Yes, but the more important thing is the imbalance. So-called "AI" can be used far more effectively and efficiently for bad.
> I have come to the conclusion that we should not allow everyone access to unlimited intelligence.
You meant unlimited information, right?
> If nine billion people all receive access to plans to build a reactor which produces unlimited energy, it just takes one religious fanatic to end the world.
is about information. No intelligence required.
Any solution to crime that starts with eliminating types of people instead of poverty is just a form of fascism.
The problem with techies is that they read a lot of SF.
We already have cheap and abundant energy tech, it's called "solar panels and batteries". And the bottlenecks to both can't be solved by Claude or Kimi, unless Claude and Kimi pick up shovels and welding equipment.
But I've also read about retrofuturism.
If you want to live far into the future, 20 years from now what you will and imagine looks very quaint, because the future never plays out like we think it does.
OpenAI is a lot closer to bankruptcy than to ASI.
I'm perfectly fine with this prediction aging badly.
Consider this 1926 government report:
https://www.derekthompson.org/p/america-1926-an-absurdly-dee...
"The authors of Recent Social Trends were astonishingly prescient about the direction of technology."
There have been many successful predictions, e.g. the internet was predicted, AI was predicted, moon landings were predicted, etc.
>OpenAI is a lot closer to bankruptcy than to ASI.
We can't count on an OpenAI bankruptcy to save us.
I'm not that optimistic, though. Either people will control AI; or people will be destroyed by AI. I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike.
There are horrifying acts that have occurred since humanity. Whole decades lost to war an millions dead, and Sep 8, 2026 is your example of "horrifying enough to live in a world"?
I think some perspective is needed.
I am also worried about this. One ray of sunshine here is that it's not a great explanation for the Fermi paradox. We would see AI civilizations all over the galaxy if this were a regular occurance. I guess that just leaves us with the "horrifying" scenario: Geoffrey Hinton's vision for our best case scenario: a paternalistic super-intelligence which saves us from ourselves.
The reason I consider it horrifying is that giving up our agency to a God-like creature (which we created) is the basis for so many horror and dystopian tropes. It's so easy for such a reality to go wrong. For example, in order to maximise aggregate wellbeing, it would be necessary to hurt individuals. Executing people who are burdensome to society. Fat people. The disabled. The elderly. The creative ways a super-intelligence might interpret their moral obligations to us could very quickly become nightmarish. And if we have no off switch, there is no escape. Ever.
[citation needed]
With all due respect, I think this statement is false.
If you live in a society that can, thanks to advanced technology, provide for the material needs of all, which, if you live in a rich Western country, is basically already the case (remember the last time you had to actually grow your own food? Probably several generations ago), you don't need to execute anyone.
Indeed, given that in such a society, where there are no shortages of food leading to "only one of us can eat" type scenarios, I'd argue that hurting individuals would lower aggregate wellbeing, because you're hurting people beloved by other people (the elderly? Turns out their grandchildren really like them and get upset if they die? fat people? Turns out their friends really like them and get upset if they die. the disabled? Turns out their family really like them and get upset if they die).
This also illustrates the messy nature of comparing social good and bad. Who am I to argue that the aggregate sadness created by removing a psychopath from society is better than a homicide? Maybe the person murdered was disliked by everyone and had no friends?
I remain resolute in my premise, however, and there is a lot of theory on the issues of aggregate social good vs individualism. John Stuart Mill's On Liberty (1859) is an excellent foundation for this. This is a subjective problem to solve, meaning that the balance between individual freedom and aggregate wellbeing is different for everyone. An AI overlord would make decisions every day which individuals would disagree with, at least at the margins.
It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).
A plague can easily be made by a lone actor, made in a home lab, and spread via airflight to a dozen locations by the same.
Gene sequences can ever be ordered online.
There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.
There is no reason to assume that super intelligent evil AI with magic powers will appear; while we are purely in the realm of (hackneyed) fantasy, why stop at imagining just one thing? You can build the whole story, not just the basilisk.
The parent comment is a story, not a prediction; it should be treated like one.
By shutting down open-weight models? Why not just do it now then?
You can do at home gene editing with open source software and have it synthesized into a bacteria for the cost of a nice meal for two (under $100), or viral vector for less than $500. That's in reach of anyone that can snatch a purse.
> That's in reach of anyone that can snatch a purse
I went to primary school with some guys who could/would snatch purses. I *promise* you, they are not able to gene edit organisms with FOSS.
I get your general point, but I think the bar to entry is still much higher than petty crime and larceny.
Using CRISPR to modify existing bacteria is probably cheaper, but still complex.
[1] https://briefglance.com/articles/elegen-slashes-dna-synthesi...
For now in most parts of the world it's easier to acquire a gun either legally or on the "grey market" and be assured that it will work for the intended purpose than to 3D print a gun, find a shooting range to test out the gun and iterate until it works fine enough.
In most part of the world, bullets are illegal. If you are getting "grey market" bullets anyway, it is easy to get a "grey market" gun.
Not only the quantity of people who have the minimum aptitude required, specialized expertise requires both knowledge and experience doing these tasks. Using an LLM requires neither.
Leaning on an LLM to do much or all of this means it can happen in seconds/minutes/hours, the LLM can do it several times during that psychotic break (as opposed to a fraction of a hack in a single episode). The barrier to entry pre-LLM was both high and the population who could pull it off (before the Chinese/Russians turned this into commerce) was low.
Hacking is an VERY asymmetric activity (the attacker only needs to "be right" once, whereas the defender has to be right every time for every asset they defend). It takes geometrically / exponentially more work to defend (while keeping high availability) than it does to defend. The more widespread tools to find vulns / generate exploits are, the faster the posture of the defense side falls from "maybe we can stop most hacks" to "we know we will fail to prevent most breaches, so we need to prioritize securing only the most valuable resources". That's a BAD place for the average company to be in.
There is the worry of the old saying "they (the attackers) only have to succeed once to win, we (the defenders) only have to fail once to lose.". In that sense there is a big imbalance, but the emergence of AI does not really affect that because it strengthens both sides.
With physical security like things like pipe bombs that's a lot more imbalanced.
However what can we do? The only effective measures include monitoring everyone which is not a solution because it will make the world not worth living in.
I'll bet it has happened at least once.
https://news.ycombinator.com/item?id=49594750
There will be many species traitors, and depending on how things play out, it could only take one.
If there is to be a world-destroying event, it will be triggered by human fear, greed, and aggression.
(I also think people massively overstate schizophrenia as an attack driver)
Thank you for mentioning it
That's a terrible framing. By this point it should be very easy for someone with 150 IQ but in reality they have issues finding someone to date.
Are 95% of worldwide terror attacks done by schizos?
Which _will_ manage the problem, but at what cost.
We've been rolling them for the past 3 years and nothing happened. Can we stop with this baseless fearmongering crap?
...pandemics are precedented. There's a world of difference between that and a fully hypothetical catastrophe that you can't even describe without looking and feeling silly.
Bill Gates is Epstein's pal, and not a very intelligent person to-boot. I trust his liberal musings about as much as I trust Chomsky's hysterical and discredited views concerning AI.
The HuggingFace hack fell into that category not long ago, yet it happened anyways.
In any case, how would AI regulation prevent any of these things? Shouldn't we instead focus on limiting access to bomb/bioweapon precursors and securing vulnerable endpoints, since that is the solution that saves lives regardless of how AI disseminates? Regulating AI is expressly dangerous, if it expands that societal blind spot. It'd be like regulating red mercury.
It’s an endless, positive irony spiral.
Often ease of access in the moment is all that matters. If there's a gun nearby you might shoot someone or yourself in a heated argument, but are less likely to go and find/buy one to use. Someone who's stopped from attempting a suicide will likely not try again (70%)
A bored/depressed/angry/curious person might try to build a pipe bomb if they can find out how easily, but are less likely to put in effort.
Depressed people usually don't have the energy to get out of bed so they're even less likely to think of hunting down instructions on how to build pipe bombs.
Mass media really has people being scared all the time.
That’s not what I’m saying, I’m saying there’s more likely to be gun involvement in disputes when people have easy access to guns than when they don’t.
For example, when the number of men with service arms fell by 20% the number of men killing themselves dropped by 8%
https://www.researchgate.net/publication/328554995_Suicide_b...
I think this part is important distinction. Military duty comes with training and rules and so on.
Compared to untrained persons having access.
Similar but apples and oranges
LLMs will not kill security, it will change. just like handheld high explosives likely changed a deal too somewhere somehow.
In reality, the skills needed are pretty basic, but they overlap pretty strongly with being sane and well-adjusted. And if you are, you're probably not daydreaming about mass murder. Exceptions happen, Unabomber and so on, but they're pretty rare. In any case, Unabomber probably didn't need a tutorial.
We don't want ChatGPT to become an enabler and a co-conspirator for an unhinged person, but I think the concern is overdone.
People go for conventional "exciting" threats rather then boring ones.
If you already have a magic interface, which helps you pro activly in responding to everything uncensored because you feel like 'observered' or whatever and then you spiral in a whole and that one partner encourages you and gives you helpful steps to do anything.
But i'm more worried that the internet gets a lot less save with uncensored frontier LLMs.
He used to stick to water (God, you see, needs to flood our town to cleanse it), but once he did a lot of damage to communications from going after water, and so he's added that to his portfolio.
For a relatively narrow subject area (e.g. construction of pipe bombs) the collation is minimal, and so the filtering and tailoring probably isn't that important; a novice doesn't learn a lot more from the LLM than they would have done from a few Google searches.
For a broad subject (practical creation and exploitation of software vulnerabilities), the collation is very significant and the filtering means that LLMs can empower a novice to act at a similar level as an expert.
The author too is confused, asserting that telling people how to build pipe bombs is malicious.
I personally have no need for an LLM which will readily explain how to cut up the genotype of smallpox into small chunks which can pass the screening at the bio-labs, and can be readily assembled into the real thing by a second year lab-student.
RC cars and planes existed for many decades already.