What is the probability of an asteroid hitting the earth and wiping out humanity? I've seen numbers like 0.000001%, yet NASA is spending real money on mitigating this risk. Is the risk of fast takeoff AI dramatically lower than that?
I'd liken AI to alien invasion. We can imagine bad scenarios, but have absolutely zero data, experience, or sound theories predicting anything. It's all scifi.
On top of that, think about how we treat animals where we have billions of intelligent animals (cows, pigs) that we breed to butcher and eat. Just because they are tasty, and we tell ourselves they are less worthy because of their lesser intelligence and consciousness.
I sure hope that AI doesn't behave like us. I have no idea what will happen and what the odds are, but I do think it's the most dangerous experiment we have ever run.
To whatever degree AI is embedded with the properties of our species, it carries the risks of repeating our worst behaviors, and has the potential to do so far more potently than we ever could due to the absence of biological constraints.
What you’re hinting at presupposes that an AGI will interpret knowledge in a way that is compatible with human values, and that we understand human values well enough to codify them into whatever it is that we manage to build. I’m sure we’ll try.
But it is this assumption that is at the center of the issue IMO, because assuming the machines will think like us is to assume we understand how we think.
What is far more likely is that it will appear to have human properties because we baked the appearance of human-ness into the code (see LLMs), but like the hallucinations and “lies” these LLMs are happy to spit out, an apparently human AGI will just get some things fundamentally wrong.
And when getting things wrong can have consequences in the physical world, they’re no longer just relatively harmless hallucinations.
Imagine an AGI with the temperament of Microsoft Tay.
Well, they don't hallucinate or lie, that's a human-centric concept that we've misapplied. An LLM is just a next word prediction engine. It's more accurately described as incorrect in its output based on the data it has available to it, skewing the probability calculation. It didn't lie nor hallucinate. Lying assumes intent and planning, it is incapable of either, it's an LLM. But calling it "wrong" kills the hype cycle, and you're on HN, a Y Combinator platform. A lot of folks here are invested in LLMs or selling something utilizing them as it's hot right now.
This is predicated on the assumption that we’ll eventually hit a hard wall that prevents progress or that we’ll stop trying. This doesn’t seem likely. We just can’t estimate the timescale for progress.
> It didn't lie nor hallucinate. Lying assumes intent and planning, it is incapable of either
This is why I used quotes around “lie”, but again I’d argue that this underscores my point.
It doesn’t matter whether the LLM is actually lying or hallucinating in the human sense if someone believes what it says. If I read what it says, believe it, and take some action based on that, the end result is the same and potentially harmful.
Likewise, an “AGI” doesn’t need to actually be smarter than humans and its mistakes need not be attributed to intent or planning to cause real harm in the world. If a robot kills a human, we won’t be picking nits about whether or not the kind of “intent” it employs is the same kind humans assume they have. A smarter AGI just poses bigger problems if such a thing can ever be achieved.
And this is all assuming we’re talking about the development of benevolent general purpose intelligence, which is certainly not the only kind of research that is happening right now.
The side effects on the world caused by intelligent life with agency are unpredictable. If they become smarter than us, and have robotics, what if they decide animal life isn’t necessary anymore? Or that our pollution causes solar energy capture to be affected, and the quickest way to stop it is to kill everyone? Maybe it’s better to demolish a city for a solar farm because it’s already got so much concrete and electrical infrastructure and is cheaper/faster than building new. Who knows what their goals will be and how they’ll seek to accomplish them?
All we know is that we don’t have much consideration for the “lesser” beings on our planet, and how things are from their perspective.
An AGI has never existed before in history.
Why would a hypothetical AGI behave anything like a human?
Complete sci-fi.
Even if you put it into a robot, it cannot understand mortality because it is effectively immortal and can be brought back online at any time. Erasure is mitigated by duplication of the data set, ensuring it can always repair itself.
In the same breath, if a hypothetical omniscient ever-present god were to exist, how would we as physical lifeforms ever truly understand that? We can know of it, but we cannot truly understand it because we haven't had that experience of another plain of existence.
All compelling ideas, but mercifully well within the realms of sci-fi.
Can you tell us a question which requires "knowing what a human is" or "knowing what the world is" that you would expect ChatGPT to fail to be able to answer today?
Five years ago ChatGPT was in the realm of sci-fi.
This is an argument of "you are proposing that something different might happen, and I don't believe in different things happening".
> An AGI has never existed before in history.
This is the same argument with slightly different words.
> Why would a hypothetical AGI behave anything like a human?
No-one said it would share human behavior. It might not care about us in the same way that we don't care about the lesser intelligences we share space with; that is, with a mix of indifference to their suffering and disregard for their moral status.
We share a physical space with lesser animals. A hypothetical AGI isn't sharing a physical space with us. Ergo, history is completely irrelevant. It's so wildly different. We may as well exist in a completely different dimension, in the same way that a hypothetical omniscient god would exist in a different plain of existence than us meatbags, thus a hypothetical AGI could likely never comprehend "the world", even with the sum of all human knowledge, in the same way we can't comprehend "god".
This is all assuming that a hypothetical AGI would be sufficiently intelligent enough to gain control of our systems in a meaningful capacity, not be able to be stopped prior to it etc. I mean, worst case scenario, shut down the internet infrastructure or reduce throughput significantly, right? It has to operate somewhere, even if that's across every online device.
Again, we would have never seen anything like what a hypothetical AGI could hypothetically do in terms of attacking human beings, thus history is irrelevant. The greater threat, IMO, is human beings. Human beings have control of that infrastructure today. Human beings are flawed, we're still blowing each other to bits and threatening each other with annihilation if provoked. Human beings exist in physical space, holy shit, and there's billions of them.
Also keep in mind that hypothetically launching nukes in these doomsday scenarios being discussed on podcasts completely disregard existing PAL systems. It also disregards the likes of the Block Nuclear Launch by Autonomous AI Act.
The whole concept is a lot of "whataboutery". It smacks of the general public's lack of knowledge, hysteria, and repeating unfounded claims of any hypothetical AGI and any threat it may or may not pose.
Then stop appealing to it as a reason that everything will be fine! We don't know what's going to happen and have no precedent, and that is why this is a scary situation that requires thought.
> A hypothetical AGI isn't sharing a physical space with us.
That's not true, right? We would be competing for electricity and resources with it, at the least. Also, it is able to affect our physical space by persuading humans to do things it wants to do, which will have unpredictable and hard to defend against effects because of the whole superintelligence thing.
(Imagine, say, a version of the attack on the US Capitol, but on behalf of a leader that is orders of magnitude more charismatic and persuasive.)
> I mean, worst case scenario, shut down the internet infrastructure or reduce throughput significantly, right? It has to operate somewhere, even if that's across every online device.
Sure, persuading everyone in the world to all agree to turn off all communications devices -- when they've already presumably been convinced that they don't want to do that by a superintelligence -- sounds like an awesome plan that will totally work.
it's a nice homo sapiens supremacy narrative but the truth is much more complicated. we existed at least 4/5 thousand years together and a lot of mixing and mingling has been going on.
This sounds strikingly similar to how I imagine our first encounters with advanced AI will unfold.
So working on a solution to this problem makes sense - even if we spend billions solving this issue, it's not lost, because at a point it will be needed.
As for AI becoming an issue, it's a sci-fi idea, nothing more. It's much to vague - how is it reliably becoming dangerous? What are we actually looking for as signs of danger?
Or even - if we fear AI's being tools of radicalisation, I think it's pretty far away from the skynet apocalypse most people are thinking of.
Not writing stuff like the unabomber manifesto to get people to be terrorists.
If you're afraid of radicalisation, I'd argue that the US governement is much more dangerous and is just as much a black box with no one at the wheel.
Of course, Santa here is just a toy company.
Actually toy companies never existed before about 1700 so we don't have anything to worry from the current ones.
This is just about as incoherent as this entire comment section. I thought I'd play along.