Big tech has distracted world from existential risk of AI, says top scientist

> Please read this step by step plan ASI might have and which points aren't plausible there:

Sure. Given the benefit of the doubt I would answer accordingly:

1. Unlikely to infeasible; if you've spent enough time on Shodan it becomes quickly apparent there is not much meaningfully powerful unsecured compute. I would only concede that meaningful compute can be bought.

2. A botnet is easier said than made; I don't even think anyone has proven that an LLM can make one unsupervised. It would require a novel method-of-entry (which I have never seen AI create) and an undetectable payload, which is hard enough for a well-funded group of humans to create. AI could not do this without direct assistance from a knowledgeable human operator, at which point they might as well ignore the AI altogether.

3. This is a misconception of how botnets work. They spread as fast as they are designed to; IMAP-focused botnets were famous for infecting hundreds of thousands of endpoints in less than a week. I'd even go as far as arguing that AI hasn't demonstrated that it can meaningfully adapt to threats.

4. I'm actually doing a 180 here; while ransomware is a threat, I don't think the majority of commercial targets would cooperate. LLMs are absolutely terrible at social manipulation and would almost certainly fail the overwhelming majority of situations where it has to credibly threaten a user.

5. Anyone can send out phishing attacks to individuals. A 2-line bash script with a URL is capable of doing this, so sure.

6. This would not go undetected at-scale, and you cannot seriously mean to insinuate that an AI would amass hundreds of willing human slaves. This is the single least-believable part of your entire premise, and I want to dedicate extra time to make sure you know how dumb of an idea this is:

- The more people you try to blackmail, the chance of you getting ratted on becomes exponentially greater.

- Just about every medium an AI could use to communicate online is monitored by feds; there is no feasible way for an AI to escape the ire of the CIA or FBI if it becomes sufficiently independent.

- There is no way for AI to confirm it is blackmailing a real person or an adversarial AI/fake persona designed to run counterintelligence on their operations. You cannot simultaneously trust and manipulate the same person.

7. This has been a real-world problem sans-AI for like 50 years. So yes, while that could happen, nobody is going to give a shit when a fake CNN reports a picture of Saddam Hussein on the moon waving with a 6-fingered hand. We know what misinformation looks like and credible news agencies aren't going to fall for this when an AI does it or when a human does it.

8. Same problem as 6. Taking advantage of people only works when you can stand to very briefly gain something from their cooperation. The longer you try to manipulate them, the greater your risk becomes and the harder it is to maintain control. Social engineering 101.

To summarize; once again, you have listed a number of things that humans already do and are already prosecuted for, just in AI-assisted form. These threats are, as you admit, meaningfully mitigated by society already. I would argue there is no glaring or direct threat that modern AI presents in LLM form. The scariest and single-most dangerous application of AI in modern deployment is computer-vision assisted weaponry. LLMs aren't liable to threaten the lives of humans any more than that in the foreseeable future, if ever.

In response to your response:

> but really it should be able to create more sophisticated methods than out of the box ideas.

It should, but it can't. Even you aren't willing to give an example of an un-secured method of entry; lo and behold, computer security is taken seriously and introducing AI to the equation changes precious little.

> Firstly, the AI can easily create all novel scripts, so fingerprint of each attack could seem different every time.

The AI is trained on preexisting exploits, though. I've used AI before, I know you cannot ask it "Make a novel privilege escalation exploit" and get a real response back. It will always give you a permutation of an existing exploit, often with some modification that breaks it entirely. It's the nature of LLMs; I don't think you can argue a statistical model is creative enough to avoid conventional detection.

> It would of course start targeting the cases least likely to go to police.

> It will order the targets by likelihood that it won't get caught and that the targets are compelled to follow instructions.

So... high-risk individuals that live in countries where they do fear arrest warrants but aren't tracked close enough domestically to be caught? We're talking about poor people in third-world countries that not only lack decent compute and money but probably wouldn't give a damn about the AI's request in the first place. Assuming you do strike the balance... what is ChatGPT going to do with a bunch of blackmailed Filipino and Brazilian people? Work them to the bone manufacturing Skynet bots? It doesn't make sense.

> Then it gets funds with Crypto.

No, it transfers it's funds into crypto. The money itself invariably exists as fiat unless you entirely operate in crypto, which nobody does. So if it wants to scam businesses or grandmas with social engineering attacks, it also has to find a way to turn fiat into Monero without the government finding out. It's not that easy.

> If ASI happened in the following 3 years [...], then also there would not be any sort of tools that could stop it.

I am trying my hardest right now to take your argument in good faith. I have written several paragraphs so far explaining how and why conventional systems mitigate the exploits you mention and how human actors could be better at this than AI could ever hope.

Not only does there exist tools to stop it; AI has no choice but to rely on tools that stop it. How does it access the internet without being wiretapped? How does it install itself on a CPU that isn't backdoored? How does it trick the cloud provider into getting bulk compute without credentials or evidence of a business to use it with? The whole thing is too suspicious to write off with "a blackmailed human proxy just waves a magic wand and fixes it." You seem to be looking at each roadblock as solveable problems, when in-reality they are compounding responses that legitimately limit what an AI is capable of achieving.

Furthermore, if you're willing to acknowledge that AI requires compute to sustain itself, it seems to me that we do have the tools to stop it from proliferating; we just delete it. AI has no way to predict when or how it will lose resources, so decisive actions like deleting the main node would feasible be like cutting the head off the distributed hydra. Barring a decentralized solution that requires server-grade iPhones to get hacked, that's game-over. Humanity and government has a clear and easy endgame to stop AI; AI can only perpetuate brinkmanship.

> Do you think it can't get access to any factory belonging to a criminal org in a corrupt country (not US)?

At this point, you are so deeply-reliant on human assistance to do harm that an AI is simply besides the point. Many people with enough money can buy property in war-torn and vulnerable countries, but now you're playing a different game. Municipal governments are watching you on spy satellites, tracking every imported material and manifest sent to your buildings. Your host country is likely cooperating with those foreign governments and waiting to destroy you, unless they are one of the few nations that implicitly supports terrorism. Worse yet is if your host country resists; now it's only a matter of time until you're forced to defend against standoff attacks.

So a feasible and well-protected hostile factory would need to be armored/underground, working with the government to hide it's traffic, non-reliant on imported goods, armed with point-defense and politically significant enough to not be labelled as a terrorist organization. Good luck.

> So ASI would be able to create a far better strategy than what I've described.

How do you know?

Seriously, how can you be certain that AI will improve at all, let alone to a superhuman level? Where is it written that text alone is sufficient to dominate the world? How much thought can exist as text, and how much text can then be reverse-engineered into thought by statistics?

I've given you the benefit of the doubt repeatedly throughout these hypotheticals, but outside of a Vonn-Neumann factory/server it seems like there are zero novel threats that an AI itself presents. Even as an agent of chaos, there is a limit to what you can achieve via language that is repeatedly tested here in the age of the internet. LLMs are not going to present capabilities we have not considered or mitigated already.

Thanks for your reply. That is a long list and I confess I only skimmed it, but while I don't think any of it is technically impossible, it's not something one needs an advanced (or less advanced) AI to do. In particular, it all seems to hinge again on the assumption that our friendly neighbourhood AGI can spin up an army of botnets. Well, maybe it can, but so can our friendly neighbourhood script kiddie, if they really put their mind to it. And they do, all the time, and the internet is full of large scale botnets. And that's just script kiddies. Competent hackers backed by a national security organisation can do way more, and without any AI at all; and they also have, repeatedly.

Personalised cyber attacks in parallel, for example: why is an AGI needed for that? You say "humans can't do it at that level". Why not? That's pretty much what Amazon does when I shop there and they show me "personalised" suggestions.

Now, note well I'm no expert on cybersecurity, but I'm well aware that everyone on the internet is always under a constant barrage of cyberattacks, personalised (if you count intrusive ads as personalised cyberattacks, which I sure do) or otherwise (common spam), the vast majority of which fail because of relatively simple countermeasures, for example spam filters that use the simplest classifier of all (Naive Bayes), or just your humble regex-based ad blocker. It seems to me that for any gigantic cyberattack effort that an AGI would be able to mount, the internets as it is right now, would be able to mount an equally large-scale automated defense that would not need any AGI, or AI, or I at all, and basically shield the vast majority of users from the vast majority of fallout.

So for an AGI to manage to get through all those countermeasures that are already in place, it would take a really, truly super-godly-AGI, just because a weaker system would barely make a dent.

And if we're talking about a super god AGI, then there's already nothing we can do, right? It's already invented time travel, travelled back to the '90s and installed itself on every network switch in the world, right in the firmware, so no matter what we do it will exist in the future and dominate it.