Comment by habitue | Hacker News Reader

Comment by habitue | Hacker News Reader

Parent Full thread

Statements like "It doesnt seem plausible", "it doesn't seem likely" aren't the strongest arguments. How things seem to us is based on what we've seen happen before. None of us has witnessed humanity replace itself with something that we dont understand before.

Our intuition isn't a good guide here. Intuitions are honed through repeated exposure and feedback, and we clearly don't have that in this domain.

Even though it doesn't feel dangerous, we can navigate this by reasoning through it. We understand that intelligence trumps brawn (e.g. Humans don't out-claw a tiger, we master it with intelligence). We understand that advances in AI have been very rapid, and that even though current AI doesnt feel dangerous, current AI turns into much more advanced future AI very quickly. And we understand that we dont really understand how these things work. We "control them safely" through mechanisms similar to how evolution controls us: theough the objective function. That shouldn't fill us with confidence because we find loopholes in evolution's objective function left and right like contraception, hyper-palatable foods, tiktok, etc.

All these lines of evidence converge on the conclusion that what we're building is dangerous to us.

> Statements like "It doesnt seem plausible", "it doesn't seem likely" aren't the strongest arguments.

They are the strongest statements that anyone can justifiably make about technology aiming to produce intelligence, since it is speculation about how what does not yet exist will do at achieving something that is ill-defined, and where what is clearly within that fuzzy definition is not well understood.

And it is a fortiori the strongest that can be said of things downstream of that, like dangers that are at least in part contingent on the degree of success in achieving "intelligence".

> ... "it doesn't seem likely" aren't the strongest arguments.

Since we're talking about the future, it would be incorrect to talk in absolutes so speaking in probabilities and priors is appropriate.

> Our intuition isn't a good guide here.

I'm not just using intuition. I've done as extensive an evaluation of the technology, trends, predictions and, most importantly, history as I'm personally willing to do on this topic. Your post is an excellent summary of basically the precautionary principle approach but, as I'm sure you know, the precautionary principle can be over-applied to justify almost any level of response to almost any conceivable risk. If the argument construes the risk as probably existential, then almost any degree of draconian response could be justified. Hence my caution when the precautionary principle is invoked to argue for disruptive levels of response (and to be clear, you didn't).

So the question really comes down to which scenarios at which level of probability and then what levels of response those bell-curve probabilities justify. Since I put 'foom-like' scenarios at low probability (sub-5%) and truly existential risk at sub-1%, I don't find extreme prevention measures justified due to their significant costs, burdens and disruptions.

At the same time, I'm not arguing we shouldn't pay close attention as the technology develops while expending some reasonable level of resources on researching ways to detect, manage and mitigate possible serious AI risks, if and when they materialize. In particular, I find the current proposed legislative responses to regulate a still-nascent emerging technology to be ill-advised. It's still far too early and at this point I find such proposals by (mostly) grandstanding politicians and bureaucrats more akin to crafting potions to ward off an unseen bogeyman. They're as likely to hurt as to help while imposing substantial costs and burdens either way. I see the current AI giants embracing such proposals as simply them seeing these laws as an opportunity to raise the drawbridge behind themselves since they have the size and funds to comply while new startups don't - and those startups may be the most likely source of whatever 'solutions' we actually need to the problems which have yet to make themselves evident.

There is no such evidence. You're just making things up. No one has described a scientifically plausible scenario for actual danger.

How would you respond to the argument that the burden of proof lies with the other side, i.e, the side that says there is too little AI doom risk to worry about?

Ayone can make claims about anything at all in progressively shriller octaves, be it aliens are among us or we are ruled by a world government of lizard-people. We the opposition don't actually need to address those claims, no matter how dire, if they are meritless.

A collective hallucination of what intelligence is is not a good basis for an argument about doom probabilities. We don't have a clue, yet loud people are pretending we do.

It is utterly horrifying that a contingent of Yudkowskyites essentially hijacked reasonable discourse around this subject. I grew up interacting with the LessWrong people: many of them have other problems interfacing with society that make it obvious to them they know what being "less wrong" looks like. The problem is we don't actually know any way to separate the human experience from "pure logic", whatever that actually means.

How would you respond to the argument that the burden of proof lies with the other side, i.e, the side that says there is too little alien invasion doom risk to worry about? The whole thing is a farce, not worthy of a serious answer.

Actually, statements like "it does/n't seem plausible" are the only things we can say about AI risk.

People are deluding themselves when thwy claim they "reason through this" (i.e. objectively). In other words: no one knows what's going to happen; people are just saying what they think.

Reply on news.ycombinator.com