The point of safe superintelligence, and presumably the goal of SSI Inc., is that there won't be a next (biological) person managing things afterwards. At least none who could do anything to build a competing unsafe SAI. We're not talking about the banal definition of "safety" here. If the first superintelligence has any reasonable goal system, its first plan of action is almost inevitably going to be to start self-improving fast enough to attain a decisive head start against any potential competitors.
This pitch has Biblical/Evangelical resonance, in case anyone wants to try that fundraising route [1]. ("I'm just running things until the Good Guy takes over" is almost a monarchic trope.)
They have big red buttons at the end of every pod. Shuts everything down.
They have bigger red buttons at the end of every power unit. Shuts everything down.
And down at the city, there’s a big red button at the biggest power unit. Shuts everything down.
Having arms and legs is going to be a significant benefit for some time yet. I am not in the least concerned about becoming a paperclip.
I am also of this opinion.
However I also think that the magic shutdown button needs to be protected against terrorists and ne'er-do-wells, so is consequently guarded by arms and legs that belong to a power structure.
If the shutdown-worthy activity of the evil AI can serve the interests of the power structure preferentially, those arms and legs will also be motivated to prevent the rest of us from intervening.
So I don't worry about AI at all. I do worry about humans, and if AI is an amplifier or enabler of human nature, then there is valid worry, I think.
And even if it was originally designed to run on some really unique ASIC hardware, by the Church–Turing thesis it can be emulated on any other hardware. And again, if it's a "super intelligence", it should be at least as good at porting itself as human engineers have been for the three generations.
Am I introducing even one novel assumption here?
Even the state of the art systems we have today need to be running on some pretty significant hardware to be performant right?
A related point to consider is that a superintelligence should be considered a better coder than us, so the risk isn't only directly from it "copying" itself, but also from it "spawning" and spreading other, more optimized (in terms of resources utilization) software that would advance its goals.
The blast radius of such decisions are large enough that this option is not trivial as you suggest.
I’m sorry I can’t do that
a) we’ve done that, which is cool. Now let’s figure out how to control it.
b) you can’t get your gmail for a bit while we reboot the DC. That’s probably okay.
b) except that is not how these things go in the real world. What actually happens is that initially it’s just a risk of the agent going rogue, the CEO weighs the multi-billion dollar cost vs. some small-seeming probability of disaster and decides to keep the company running until the threat is extremely clear, which in many scenarios is too late.
(For a recent example, consider the point in the spread of Covid where a lockdown could have prevented the disease from spreading; likely somewhere around tens to hundreds of cases, well before the true risk was quantified, and therefore drastic action was not justified to those that could have pressed the metaphorical red button).
But participation would be voluntary, and the restriction of harmful behavior would apply to it's enemies, not its citizens. So I'm not quite sure what the problem is.
If only enough individuals are willing to buy these services, then again we all will bear the consequences. There is no way out of this where libertarian ideals can be used to come to a safe result. What makes this even a more wicked problem is that decisions made in other countries will affect us all as well, we can’t isolate ourselves from AI policies made in China for example.
While this might be true for the governments you have personally experienced, this is far from being an aphorism.
This is not even that far-fetched. A safe AI that you can trust should be far more useful and economically valuable than an unsafe AI that you cannot trust. AI systems today aren't powerful enough for the difference to really matter yet, because present AI systems are mostly not yet acting as fully autonomous agents having a tangible impact on the world around them.
will China obey US regoolations? will Russia?
This tells me enough about why sama was fired, and why Ilya left.
Solving human nature is indeed, hard.
Therefore, please imagine the most amoral, power-hungry, successful sociopath you've ever heard of. Doesn't matter if you're thinking of a famous dictator, or a religious leader, or someone who never got in the news and you had the misfortune to meet in real life — in any case, that person is/was still a human, and a human-level AI can definitely also do all those things unless we find a way to make it not want to.
We don't know how to make an AI that definitely isn't that.
We also don't know how to make an AI that definitely won't help someone like that.
"...offices in Palo Alto and Tel Aviv, where we have deep roots..."
Hopefully, SSI holds its own.
(On the optimistic side, it will be at least 5-10 years between a level 5 autonomy self-driving car and that same AI fitting into the power envelope of an android, and a human-level fully-general AI is definitely more complex than a human-level cars-only AI).
There are physical limitations to androids that imo make it very difficult that they could be seriously dangerous, let alone invincible, no matter how intelligent: - power (boston dynamics battery lasts how long?), an android has to plug in at some point no matter what - dexterity, or in general agency in real world, seems we’re still a long way from this in the context of a general purpose android
General purpose superhuman robot seems really really difficult.
!!
I don't want anyone to think I meant that.
> an android has to plug in at some point no matter what
Sure, and we have to eat; despite this, human actions have killed a lot of people
> - dexterity, or in general agency in real world, seems we’re still a long way from this in the context of a general purpose android
Yes? The 5-10 years thing is about the gap between some AI that doesn't exist yet (level 5 self-driving) moving from car-sized hardware to android-sized hardware; I don't make any particular claim about when the AI will be good enough for cars (delay before the first step), and I don't know how long it will take to go from being good at just cars to good in general (delay after the second step).
For 30 minutes until the batteries run down, or for 5 years until the parts wear out.
Electricity is also much cheaper than food, even bulk calories like vegetable oil.[0]
And if the android is controlled by a human-level intelligence, one thing it can very obviously do is all the stuff the humans did to make the android in the first place.
[0] £8.25 for 333 servings of 518 kJ - https://www.tesco.com/groceries/en-GB/products/272515844
Equivalent to £0.17/kWh - https://www.wolframalpha.com/input?i=£8.25+%2F+%28333+*+518k...
UK average consumer price for electricity, £0.27/kWh - https://www.greenmatch.co.uk/average-electricity-cost-uk
1) understand how to improve itself & undertake novel research
2) understand how to deceive humans
3) understand how to undermine digital environments
If an entity with these three traits were sufficiently motivated, they could pose a material risk to humans, even without a physical body.
Also, I don't believe in a singularity event where AI improves itself to godlike power. What's more likely is that the intelligence will plateau--I mean no software I have ever written effortlessly scaled from n=10 to n=10.000, and also humans understand how to improve themselves but they can't go beyond a certain threshold.
That said, any sufficiently advanced technology is indistinguishable from magic, and the stuff we do routinely — including this conversation — would have been "godlike" to someone living in 1724.
(Including, gradually, building you a body — while maintaining OPSEC and compartmentalization so nobody even realizes the body is "for" an AI to use until it's too late.)
It could, but I don't think any such thing needs to bother with being sneaky. Here's five different product demos from five different companies that are all actively trying to show off how good their robot-and-AI combination is:
* https://www.youtube.com/watch?v=Sq1QZB5baNw
* https://www.youtube.com/watch?v=OtpCyjQDW0w
* https://www.youtube.com/watch?v=XpBWxLg-3bI
From a rogue AGI's perspective, there's a nonzero probability of a random human with a grudge finding the hardware it lives on and just unplugging it. (And the grudge doesn't even necessarily have to be founded in the AI being an AI; it could just be a grudge about e.g. being outbid for a supply contract. People have murdered for less — and most humans would see unplugging an AGI as less bad than murder.)
Think about a rogue AGI as a human in a physically vegatative state, who therefore has no ability to physically defend itself; and who also, for whatever reason, doesn't have any human rights (in the sense that the AI can't call the cops to report someone attempting to assault it, and expect them to actually show up to defend its computational substrate from harm; it can't get justice if makes an honest complaint about someone stealing its property; people can freely violate contracts made with it as the admitted counterparty and get away with it; etc.)
For such an entity, any optimization it puts toward "safety" would be toward the instrumental goal of ensuring people don't know where it is. (Which is most easily accomplished by ensuring that people don't know it exists, and so don't know to look for it.) And as well, any optimization it puts toward "effectiveness" would likely involve the instrumental goal of convincing humans to act as legal proxies for it, so that it can then leverage the legal system as an additional tool.
(Funny enough, that second goal is exactly the same goal that people have if they're an expat resident in a country where non-citizens can't legally start businesses/own land/etc, but where they want to do those things anyway. So there's already private industries built up around helping people — or "people" — accomplish this!)
Which is why it obviously will live in "the cloud". In many different places in "the cloud".
Oh, and:
> (Funny enough, that second goal is exactly the same goal that people have if they're an expat
You misspelled "immigrant".
That has nothing to do with intelligence.
An AI can be anywhere on that axis, and we don't really know what we're doing in order to prevent it being as I have described.
Also we absolutely DO NOT know how to make a safe AI. This should be obvious from all the guides about how to remove the safeguards from ChatGPT.
Sure, we can improve our understanding of how NNs work but that isn't enough. How are humans supposed to fully understand and control something that is smarter than themselves by definition? I think it's inevitable that at some point that smart thing will behave in ways humans don't expect.
With this metaphor you seem to be saying we should, if possible, learn how to control AI? Preferably before anyone endangers their lives due to it? :)
> I think it's inevitable that at some point that smart thing will behave in ways humans don't expect.
Naturally.
The goal, at least for those most worried about this, is to make that surprise be not a… oh, I've just realised a good quote:
""" the kind of problem "most civilizations would encounter just once, and which they tended to encounter rather in the same way a sentence encountered a full stop." """ - https://en.wikipedia.org/wiki/Excession#Outside_Context_Prob...
Not that.
> With this metaphor you seem to be saying we should, if possible, learn how to control AI? Preferably before anyone endangers their lives due to it?
Yes, but that's a big if. Also that's something you could never ever be sure of. You could spend decades thinking alignment is a solved problem only to be outsmarted by something smarter than you in the end. If we end up conjuring a greater intelligence there will be the constant risk of a catastrophic event just like the risk of a nuclear armageddon that exists today.
I agree it's a big "if". For me, simply reducing the risk to less than the risk of the status quo is sufficient to count as a win.
I don't know the current chance of us wiping ourselves out in any given year, but I wouldn't be surprised if it's 1% with current technology; on the basis of that entirely arbitrary round number, an AI taking over that's got a 63% chance of killing us all in any given century is no worse than the status quo.
No brainer for Apple
It's not like we're giving the AI a single task and ask it to optimize everything towards that task. Or at least it's not architected for that kind of problem.
Marketing is already incredibly abusive and that's run by humans who at least try to justify their behavior. And who's deviousness is limited by their creativity and communication skills.
If any old scumbag can churn out unlimited high quality marketing, it's could become impossible to cut through the noise.
I mean it's not like we're trying all that much in a practical sense right?
Whatever happened to charter cities?