The Coming AI Hackers (2021) | Hacker News Reader

The Coming AI Hackers (2021)

schneier.com

70 pointsjosefslerka51 comments

Normally I finish Bruce's essays with a sense of clarity and having read a sharp analysis. Not saying there's anything technically amis with this one, it's all terrifyingly clear, but I think he bit off a little more that he could chew. The result is indigestible. I personally would have split this into several shorter pieces.

When it's all put together this way though, I can't help think we've shot ourselves in the foot. As computer scientists and developers we've probably already lost control.

We owe a debt of honesty to the world to say so now and stop pretending otherwise. Then we can, as a society, revisit Postman's Seven Questions:

1. What is the problem that this new technology solves?

2. Whose problem is it?

3. What new problems do we create by solving this problem?

4. Which people and institutions will be most impacted by a technological solution?

5. What changes in language occur as the result of technological change?

6. Which shifts in economic and political power might result when this technology is adopted?

7. What alternative (and unintended) uses might be made of this technology?

Thanks, I had not seen these Seven Questions before. As a pragmatist (?), I suspect answering from 3) on down is near impossible* unless you actually release it into the world. People are clever in a way I don't think any of us can anticipate and put to use things in novel ways that often surprise their creators.

* Well, at best it would devolve into arguing back and forth and neither side able to convince the other what problems would or would not be caused or are avoidable.

> As a pragmatist (?), I suspect answering from 3) on down is near impossible* unless you actually release it into the world. People are clever in a way I don't think any of us can anticipate and put to use things in novel ways that often surprise their creators.

That is likely true in many cases, but, even if definitive answers cannot be provided, the questions should still be asked in advance and the answers pondered. Many people take that difficulty or impossibility of conclusive answers as justification for not asking those later questions.

Well, you can't "idiot proof" most things, because the "idiot"s are so ingenious.

In this case though, I think it'll be the big guys who figure out how to fill the world with cheap almost humanly creative garbage. Whether it's spam, social engineering, or attention addiction they'll monetize it first and keep the little guys in mostly walled gardens of "creativity".

I feel HN has regarded blockchain and smart contracts with total derision as a solution in search of a problem. And yet with AI, it can actually harm the world but since it does impressive things, these questions are barely asked.

Progressives have started to talk about how “tech bros” think technology can improve everything but it also creates massive problems they IGNORE.

Blockchain has harmed the world. Ransomware, numerous scams, greenhouse gases... Before all of those harms, very few people had much to say about Blockchain outside of utopian visions.

AI has yet to cause harm. If/when it does, I'm sure the conversation will, likewise, turn sour.

It will be too late. These bots can be easily proliferated and attack in swarms, to infiltrate any forum shout down any group they don’t like, sow dissent, astroturf revolutions. People keep investigating how one bot will fare among a poker table of 8 players. No one seems to consider that these bots can very cheaply SURROUND all the humans 99999 to 1.

It is the nature of technology that we rarely grasp how it will change things beforehand.

But how novel is that really? Spam is already an issue. You solve it at the account level, not the message level. Trust graph forever.

Spam doesn’t solve the Turing test interactively and fool millions of your fellow users at scale.

The problem is that any answers that anyone comes up with for new technologies are completely worthless. No one knows how any technology will play out or its consequences. Read about the confident predictions people made for air power once the airplane was invented, and compare that with the reality: it's not that air power made no difference to warfare, but it made a much smaller difference than people imagined when it was first invented.

Secondarily, the people most eager to answer such questions are the people like Postman, people who are Concerned as a hobby, the people most eager to tell us that we are "Doing it wrong", and only if we listened to them we would be happy.

Do you have some sources where I can read about the wrong predictions of air power? It is a bit hard to imagine predictions overstating the importance of air power in relation to their actual usefulness in modern warfare.

This is a good primer:

https://acoup.blog/2022/10/21/collections-strategic-airpower...

The parallel with AI GP is making is very interesting. People at the time thought air power would be transformative in ways that didn't pan out, while missing more subtle applications.

AGI isn't comparable with any prior technology because we have spent the last ~million years being the most intelligent thing on this planet, and AGI seeks to upset that.

If you have to make a comparison, the most apt would be the introduction of invasive species to existing ecosystems. Any one existing species may be OK after the introduction of an invasive species to its ecosystem, but it usually ends badly and introducing invasive species is generally something we try to avoid doing to other species.

Of course AGI will not compete with us for food, but it will completely destroy our present economic system by creating vast unemployment.

Conditions of severe economic change with an unemployed population and huge amount of technological power in the hands of a few, seems ripe for producing totalitarianism.

I have nothing to reply except that as far as I'm concerned this is fanfiction.

It's a potential future. Nobody knows what will happen post-AGI. But that doesn't mean we shouldn't think about it.

I contend that people who think that it's going to be fine are probably extrapolating from their own recent past (i.e. their life has been relatively comfortable so far), rather than humanity's past, which is quite a bit darker.

> Do you have some sources where I can read about the wrong predictions of air power?

The Bombing War: Europe 1939-1945 by Richard Overy is a great and comprehensive start.

Very approximately, air power appeared in WW1 where it was essentially tactical. In the UK, the Army's Royal Flying Corps was then re-established as a post-war separate, independent entity (the Royal Air Force) and in the intervening period convinced many that a strategic bomber force could defeat Germany largely on its own. This was supported on a general hysteria about the apocalyptic power of the bomber (itself based on early SF by people like HG Wells, in which cities and morale would be devastated by fleets of bombers) as well as the imperial experience, where the RAF was used to police provinces from the air (e.g. Somaliland and Afghanistan).

However, when WW2 broke out, the effects of strategic bombing were initially negligible, with massive aircraft / crew losses for minimal strategic effect, and it took years for the bomber force to achieve the weight necessary to disrupt German manufacturing.

> answers that anyone comes up with for new technologies are completely worthless. No one knows how any technology will play out or its consequences.

The "Experts. What do they know, huh?!" thing is a little dishonest. Wouldn't you rather give Einstein the benefit of the doubt; that someone who is smart enough to design an atomic bomb is maybe smart enough to understand what it will do?

> the people most eager to answer such questions are the people like Postman, people who are Concerned as a hobby

No. It's not like these are different groups of people, one virtuous and pure in their desire to "make the world a better place" and another band of unwashed neo-Luddite primitives who jeer and throw rocks. What you're doing is called "splitting" [1]. We desperately need more people like Postman (even better if they have CS degrees too) to relieve us of the delinquent fantasy that "no consequences can be seen".

[1] https://en.wikipedia.org/wiki/Splitting_(psychology)

Interesting, but long read. To summarize: people are easy to manipulate and AIs are learning how to do this at scale and with intent. AIs are still tools though. It's easy to anthropomorphize AIs and think in terms of us and them. It's how we are wired to think. But the reality is that they are expensive tools that are owned and wielded by other people.

So, you get all the unimaginative dystopian thinking. Which is of course largely the product of science fiction written decades ago. It's hard to be innovative in this space. We already imagined all the bad outcomes a long time ago. But that doesn't mean it will play out like that. Precisely because we've imagined all the bad stuff, that's likely to not happen.

The short term reality is actually an arms race between companies and countries. And like most arms races, the people with the best toys and the most resources end up on top. The question is not what AIs will do but what the people wielding them will do with them. And how we can hack their purposes. The system these companies and countries operate in is of course hack-able. Democracies are a power hack. We once had these all powerful kings, emperors, and dictators. And then the people that gave them power got smart and organized. Democracies basically curb that power in the interest of self preservation.

Like with most technology, the answer to what will happen with AIs will be mostly harmless and beneficial stuff. That's where the money is. But with some intentionally harmful stuff and maybe a little bit of unintentionally disastrous stuff. So, we as a society need to get better at countering/preventing/disincentivizing the bad stuff and preventing the unintentionally harmful stuff. Rejecting technology is not the answer. It just makes us more vulnerable. The more of us get smarter, the better it is for all of us.

This was summarized by chatGPT*

I mean no offence at all, but it somehow feels like you don't know who the author is. Doesn't mean you cannot disagree of course, but I believe his opinion deserves credit :-).

The second thing is that I strongly disagree with your statement that technology is mostly harmless and beneficial. In one sentence: it is responsible for the current mass extinction. But of course there are tons of small examples that show how technology is generally not "mostly harmless and beneficial". I would even argue that blockchain, for instance, has turned out to be mostly harmful: I am still looking for a constructive use-case of it, though criminal ones are legion.

ChatGPT, for instance, will bring phishing/impersonation/disinformation to a whole new level. That should be scary, given the impact of less elaborate disinformation technologies in the past.

> I mean no offence at all, but it somehow feels like you don't know who the author is.

I've been reading Bruce Scheier's blog for years. And I actually don't disagree with him. Just adding my opinions on top. I don't think there's a big conflict between what the two of us are saying. I think he might even partially agree with me.

> I strongly disagree with your statement that technology is mostly harmless and beneficial.

That's your good right. But my opinion does not come from ignorance. Which is what you seem to be implying.

To counter your opinion, you only talk about the negative and completely ignore the positive impacts of technology. ChatGPT (to stay on topic) is already a big educational tool. Anyone curious enough to ask it questions is getting answers. The fact that some spammers can also use this technology is not a reason to put a stop to that.

Sorry I did not mean to imply that your opinion comes from ignorance! I just wanted to make sure you knew who the author is, which I find important and which ChatGPT (since we talk about it :)) completely removes.

ChatGPT does answer questions indeed, but a risk I see is that it teaches people not to question the origin of information (ChatGPT completely loses it). But in a world with more and more disinformation, it is a vital skill to be able to check a source.

In my opinion, a big part of education is not to get an answer from the professor (here ChatGPT), but to learn how to check information and to be critical about it. "Who wrote that? Why did they write it? Who are they actually? What do others say?" -> ChatGPT not only completely loses that, but it may actually introduce mistakes while it generates the text. I wouldn't want a teacher who routinely introduces mistakes in what they say, while using their authority to make me believe it is true.

This article is written as if scheier is going to make collective decisions on behalf of everyone, or as if someone making collective decisions for everyone is going to stop everything "because it's dangerous".

What's actually going to happen is that regardless of the so called possible dangers, there will be talented humans who still view this as a net positive thing and they will continue to develop AI, wherever it goes.

Unless there's a global traumatic event, I don't see anything stopping.

AI developers have an interest to fake moral problems to better capitalize on their product (we don't support you running models at home, it's too dangerous), and that's exactly where their ethics concerns will end.

We can't deal with any of these things by telling people to stop doing things they obviously won't stop doing.

I'd rather we deal with this transparently. How about requiring AI models deployed to be really open? Then at least humanity will always have an advantage over AI that AIs are whitebox while humans are blackbox.

And if some AI will do some hack to benefit its creators, everyone else will have a chance to understand it's happening.

It shouldn't be "this is dangerous, make it closed so only we can abuse it", it should be, this is dangerous, so we're doing this as transparently as possible.

I think in a theoretical war between Turing machines, the Turing machine which is given as input the code of the other turing machine should always be the winner.

Think about the halting problem and diagonalization: the diagonalizing counter example wins by having the source code of the program which supposedly solves the halting problem.

Or we could have a Butlerian Jihad. :)

Did you read the part about the Explainability Problem? I think it says why it is fundamentally not transparent.

> But some bugs in the tax code are also vulnerabilities. For example, there’s a corporate tax trick called the “Double Irish with a Dutch Sandwich.”

So how come financial hackers exploiting vulnerabilities in the tax code and making these sandwiches never do jail time, while computer hackers regularly do?

Computer hackers don't necessarily do jail time.

Someone hacking a single-player video game, to give them increased stats or something, won't do jail time. Even when it's a multiplayer game, someone finding a hack that benefits them is unlikely to face jail time.

Hacking is a crime if you're gaining unauthorized access to a system, especially a remote one. I think hacks involving money and personal user data have different criminal classifications as well.

And that's ignoring the massive number of hackers ("researchers") who actually get paid to do it (many of whom started out doing illegal things).

Yes, but a hacking and making millions is generally a crime, while exploiting the tax code to make millions or even billions is accepted behavior.

As if there's an implicit belief that companies artificially reducing their tax liability will lead to better growth or something? Perceived benefit is then positive, I suppose. On the other hand, an individual doing it for their personal gain is vilified indeed.

You don’t understand, when it comes to taxes we follow the letter of the law, not the spirit. Financial hackers are technically using legal means to accomplish their goals. The law is the law.

My point is that this is strange.

Breaking into a government website is illegal. However if the code on their server was part of "the law", it suddenly becomes legal?

If the law says it’s legal, that is the law.

Still strange. It should be: if it's against the spirit of the law, it's illegal.

Otherwise we could replace judges by computers who just evaluate logic statements in the book of law.

No, fuck the spirit of the law. If that’s what society runs itself on then the written laws don’t matter and the real laws become whatever subjective bullshit the judges decide at the moment. “Hidden” laws essentially. Dystopia.

Logic statements that evaluate to legal or illegal is the ideal when it comes to laws, judges are only necessary when logical certainty isn’t possible.

I see a contradiction there. You want the law to be like code, but actual computer code you are still not allowed to use in unintended ways? How would that last part be encoded in the law?

Anyway, I'm ok with a more logical, objective interpretation of the law, but then also the loopholes in the law should be treated as code exploits: it should be possible to report them and then they should be fixed asap.

The loopholes are not exploits. They are unintended consequences of poorly written laws. What you are asking is for laws to be rewritten. There are already normal procedures in place for proposing changes to laws. Write to your congressman.

The code is not the law. You break laws by breaking into a government website.

You don't break law by hacking it, that's the point. A loophole is not illegal, it's a workaround.

So if a website gives one access to some data, is it okay because it clearly serves it to public?

If you mean "exploiting" loopholes, then they are lawful and usually working as intended. I never considered my CPA a criminal hacker; I'm sure he'll be amused.

If you simply mean "not paying your taxes" then this is a crime that is frequently prosecuted and is not particularly clever or a hack.

You appear to be conflating the two for some reason.

If you mean, "unauthorized access of a computer system", then yes, that is unlawful and would possibly be one reason why some people are prosecuted and sent to jail.

Nit:

>> For example, there’s a corporate tax trick called the “Double Irish with a Dutch Sandwich.”

This has not been a thing since 2020.

> You appear to be conflating the two for some reason.

No, I'm just abstractly (like TFA) considering hacking to be an activity where a system (code or financial law) is brought outside of its intended use.

> "exploiting" loopholes

Anyway, why aren't there bounties for finding loopholes and why aren't they fixed as soon as possible?

Because the loopholes are a feature. Working as intended in the vast majority of instances.

You seem to consider that the law is defined by the system being hacked. It is not: the law is the law. The code of a website is not the law.

It is legal to hack a system if it does not break the law (I can use a fork as a screwdriver if I want to), but it is illegal if it breaks the law (I can use the fork as a weapon and break the law).

A loophole is legal. It may feel like it should not be, because it allows to work around the law. But it is legal.

> So how come financial hackers exploiting vulnerabilities in the tax code and making these sandwiches never do jail time, while computer hackers regularly do?

That's because Schneier defines hacking in the original sense of the word, subverting or misusing a system to get a desired, but unintended, result.

The computer "hackers" doing jail time are not hackers by Scheier's definition.

It seems inevitable that we will need some kind of PKE in order to verify identities. My ideal system is using the CAC protocol from the DoD at each of the 50 DMVs in the US to issue identification.

I don’t think this form of state-issued PKE should be required for getting online but I would prefer interacting with people in an environment where they were needed to participate. Of course anonymous forums should be allowed but I don’t want anonymity in every interaction…

…especially over the next few years as an increasing number of actors will be non-human…

> First, participants interacted with the robot in a normal setting to experience its performance, which was deliberately poor. Then, they had to decide whether or not to follow the robot’s commands in a simulated emergency. In the latter situation, all twenty-six participants obeyed the robot, despite having observed just moments before that the robot had lousy navigational skills. The degree of trust they placed in this machine was striking: when the robot pointed to a dark room with no clear exit, the majority of people obeyed it, rather than safely exiting by the door through which they had entered.

I see this all the time - people putting their faith in systems and rules and computer programs despite knowing that they're more than likely wrong in the given situation. It's bizarre.

DARPA had a program some years back to train AI on actual hackers to model their behaviors. Anyways, I still doubt AI's ability to cause significant harm beyond automating spear-phishing, social engineering, mass-scanning codebases/binaries and exploiting low-hanging fruits. Modern day exploits are still outside the realm of what AI can do - and I don't even believe AI will ever exceed that realm. But I think society will do a better job w/ security once AI automates and annoys everyone

>> This hacking will come naturally, as AIs become more advanced at learning, understanding, and problem-solving.

Everyone is so sure "AI"s will continue improving (not to say, everyone is so sure they have improved). It's going to be a little embarrassing if the foretold continuous AI improvement does not come to pass.

:grabs popcorn:

I do agree, though it feels like ChatGPT, for instance, already has a lot of potential to harm. It can write disinformation at an incredible scale. Now whenever you visit a website, just ask yourself this question: "was it written by a person, or by a bot?". It's getting hard to say.

Of course it does not always matter. But intuitively, wouldn't you put more trust into a website written by a human that actually understand what they wrote? "Visiting Prague in 3 days" written by a human may not be perfect, but at least the human understand the idea of "visiting a city in 3 days".

Now think about actual disinformation. Can you still put any trust in an article not written on a website you know (e.g. The Guardian)? It was already a problem before, but with AI like GPT, it is now possible to "flood" the web with disinformation written in pretty high quality language...

>> I do agree, though it feels like ChatGPT, for instance, already has a lot of potential to harm.

Absolutely, but that is because it hasn't "advanced" as much as some people seem to think. Others seem to see the problems but they seem to hope or believe that they will all go away as those systems "advance" (or "improve" or "progress") further, when they may as well just have reached a dead end after which they can only become bigger and more bloated as time goes by, and still be dumb as bricks, and the only thing that will "advance" is their ability to cause chaos and make a mess.

I like this quote by Pedro Domingos:

>> People worry that computers will get too smart and take over the world, but the real problem is that they’re too stupid and they’ve already taken over the world.

https://www.washington.edu/news/2015/09/17/a-q-a-with-pedro-...

Haha nice quote! :)

Reply on news.ycombinator.com