Our intuition isn't a good guide here. Intuitions are honed through repeated exposure and feedback, and we clearly don't have that in this domain.
Even though it doesn't feel dangerous, we can navigate this by reasoning through it. We understand that intelligence trumps brawn (e.g. Humans don't out-claw a tiger, we master it with intelligence). We understand that advances in AI have been very rapid, and that even though current AI doesnt feel dangerous, current AI turns into much more advanced future AI very quickly. And we understand that we dont really understand how these things work. We "control them safely" through mechanisms similar to how evolution controls us: theough the objective function. That shouldn't fill us with confidence because we find loopholes in evolution's objective function left and right like contraception, hyper-palatable foods, tiktok, etc.
All these lines of evidence converge on the conclusion that what we're building is dangerous to us.
They are the strongest statements that anyone can justifiably make about technology aiming to produce intelligence, since it is speculation about how what does not yet exist will do at achieving something that is ill-defined, and where what is clearly within that fuzzy definition is not well understood.
And it is a fortiori the strongest that can be said of things downstream of that, like dangers that are at least in part contingent on the degree of success in achieving "intelligence".
Since we're talking about the future, it would be incorrect to talk in absolutes so speaking in probabilities and priors is appropriate.
> Our intuition isn't a good guide here.
I'm not just using intuition. I've done as extensive an evaluation of the technology, trends, predictions and, most importantly, history as I'm personally willing to do on this topic. Your post is an excellent summary of basically the precautionary principle approach but, as I'm sure you know, the precautionary principle can be over-applied to justify almost any level of response to almost any conceivable risk. If the argument construes the risk as probably existential, then almost any degree of draconian response could be justified. Hence my caution when the precautionary principle is invoked to argue for disruptive levels of response (and to be clear, you didn't).
So the question really comes down to which scenarios at which level of probability and then what levels of response those bell-curve probabilities justify. Since I put 'foom-like' scenarios at low probability (sub-5%) and truly existential risk at sub-1%, I don't find extreme prevention measures justified due to their significant costs, burdens and disruptions.
At the same time, I'm not arguing we shouldn't pay close attention as the technology develops while expending some reasonable level of resources on researching ways to detect, manage and mitigate possible serious AI risks, if and when they materialize. In particular, I find the current proposed legislative responses to regulate a still-nascent emerging technology to be ill-advised. It's still far too early and at this point I find such proposals by (mostly) grandstanding politicians and bureaucrats more akin to crafting potions to ward off an unseen bogeyman. They're as likely to hurt as to help while imposing substantial costs and burdens either way. I see the current AI giants embracing such proposals as simply them seeing these laws as an opportunity to raise the drawbridge behind themselves since they have the size and funds to comply while new startups don't - and those startups may be the most likely source of whatever 'solutions' we actually need to the problems which have yet to make themselves evident.
A collective hallucination of what intelligence is is not a good basis for an argument about doom probabilities. We don't have a clue, yet loud people are pretending we do.
It is utterly horrifying that a contingent of Yudkowskyites essentially hijacked reasonable discourse around this subject. I grew up interacting with the LessWrong people: many of them have other problems interfacing with society that make it obvious to them they know what being "less wrong" looks like. The problem is we don't actually know any way to separate the human experience from "pure logic", whatever that actually means.
People are deluding themselves when thwy claim they "reason through this" (i.e. objectively). In other words: no one knows what's going to happen; people are just saying what they think.
What's the most plausible (even if you find it implausible) disaster scenario you came across in your research? It's a little surprising to see someone who has seriously looked into these ideas describe the bundle of them as "like Skynet."
It seems silly to me the idea of risk is all concentrated around the runaway intelligence scenario. While that might be possible there is real risk today in how we use these systems.
It is likely conditional on the price of compute dropping the way it has been.
If you can basically simulate a human brain on a $1000 machine, you don't really need to employ any AI researchers.
Of course, there has been some fear that the current models are a year away from FOOMing, but that does seem to be just the hype talking.
Based on the evidence I've seen to date, doing this part at the scale of human intelligence (regardless of cost) is highly unlikely to be possible for at least decades.
(a note to clarify: the goal "simulate a human brain" is substantially harder than other goals usually discussed around AI, like "exceed domain expert human ability on tests measuring problem solving in certain domain(s).)
Because you could easily find ways to print money e.g. curing types of cancers or inventing a better Ozempic.
But the fact is that there is no path to simulating a human brain.
It seems very obviously fundamentally solvable, though I agree it is nowhere in the near future.
I could see P=NP being impossible to prove but I find it hard to believe intelligence is impossible to figure out. Heck if you said it’d take us 100 years I would still think that’s a bit much.
Time travel. Teleportation through quantum entanglement. Intergalactic travel through wormholes.
And don't get me wrong they are hard. But just another class of problems. Right ?
Some have made this argument (quantum effects, external fields, etc.).
If any of these are proven to be true then we are looking at a completely different roadmap.
Can you please enlighten us then since you clearly know to what extent quantum effects exist in the brain.
It’s odd to say “reproduce quantum interactions” but remember to the extent they exist in the brain, they also behave as finicky/noisy quantum interactions. They’re not special brain quantum things.