Why are you so sure I'll lose?
twitter.com
twitter.com
Instead play with humans and have fun.
Meanwhile all battlefields will one-by-one become battlefields where the AI has the advantage. Because in the end almost every contest between humans is either one based on strength or on speed and the AI will always be able to match us for speed.
And strength has long been the domain of machines, couple machines with AI and you lose all of these as well and you lose them faster.
The biggest question to me is this one: "Does it matter?" Do we really care if there is an entity that is smarter than we are? Because after all all of the battlefields are of our own making. And I think the answer is yes because the first thing that tends to happen with new tech is that it is weaponized. And weaponized AI could easily translate into dominance by whichever party gets to it first.
I don't think rock climbers need to worry about Stockfish.
We're going to lose to specialized machines every time while putting to shame any general purpose machines of our own creation. At least for a while yet no machine is going to match the breadth of our capabilities and our self-sufficiency.
It's no surprise that something that is optimized for being capable of many things can get beaten by a thing optimized for a single purpose.
So when to start worrying?
Of course the professional competitors don't have to worry about competing against a human that's cheating using a robot; the exoskeleton would be obvious.
But it isn't unimaginable that, in some distant future, nanobots could be inject to infiltrate our muscles and tendons, and then assemble in to fibers that would assimilate with them and look normal from the outside. Competitions are won by incredibly small percentages.
I expect it to be an agent capable of independent action(even if limited by whatever restraints we put on it) it should be able to want to do things rather then just waiting around idling waiting for human input. It should be always learning, rather than only learning in batches when we tell it to. It should be able to encounter completely new forms of input/problems/scenarios and start learning on its from that and actually use concepts learned from other forms of input to learn faster in new inputs. Like for instance, if I grab an LLM and train it on an image set that doesn't contain any cars, but contains other vehicles, when I show it a car will it know that it's a car on the first shown image because it already knows what a wheel is from the other images and know that a car has 4 wheels from its language model even though it's never seen a car? It should be able to connect previously learned concepts like that. For that matter it shouldn't need absurd amounts of data to learn a new example after it has gained some basics on that specific form of input, like if it knows what cats and dogs are it shouldn't need thousands of examples of rats to know what a rat is(I'm assuming it was trained in a generalist dataset so it starts learning about the real world, it just never saw a rat specifically). It should be able to do this stuff for arbitrary stuff consistently. It should be able to explain its reasoning when asked why it did thing X. It should be able to ask questions when in doubt and learn from singular answers. Assuming it has already learned fundamentals of the real world like language and video It should be able to make predictions of the real world from learning from raw real world data with no human processing.
My impression is that a lot of these requirements are still very much unachievable and more importantly they probably won't be fixed by just making a single more powerfull ANN, they require whole new ideas to figure out, which we might figure out tomorrow or we might figure out 50 years from now or 100, we don't know, it will happen when it happens, in which case I'm defaulting to longer timeframes because I'm a pessimist I guess :). I realize that you can reduce the bar for AGI a lot to make it easier to achieve but that just makes the jump from AGI to ASI even larger. And my complaint was ultimately not specifically about AGIs I just used them as indicator of how unattainable ASI still is. If we are having problems with AGI what hope do we have of figuring out ASI on a short timeframe? And I mean ASIs are the real threat that the doomers complain about so much. You can't "just turn off" an ASI but you should definitely be able to "just turn off" an AGI.
Sorry for the long post hope this helped explain my point of view on the issue.
- Independence + continual improvement (it's always either doing something useful whether you tell it to or not)
- Reliable, prompt and accurate learning of new material
The first one is interesting because it's theoretically possible to try right now (just put an LLM scanning in a background job), but clearly not trivial.
But I think the second issue you raise is the more fundamental one. There have been a couple papers I've seen that have demonstrated that most major LLMs struggle with recalling things in a way that makes use of equivalence and substitution -- for example, if I tell GPT that X is Y's son, then if it will know that if I ask who is the father of Y, it understands the answer is X. If it cannot show basic competence in equivalence and substitution, then it begins to break through the 4th wall of a "sentient entity" and phenomenologically reveal its true colors as a stochastic parrot.
It's fascinating. There are conceptual issues, as you mention, and they're not just theoretical, but they're very noticeable (for now).
Here's the question I would ask you -- do you think even if we're no where near AGI, the world has fundamentally changed since the mass distribution of LLMs? I have felt a noticeable shift as the technology's adoption is beginning to alter how society conceives of and digests media. It feels like a discrete new evolutionary state of the internet. Social media fabric feels like it's in a new phase compared to even a year ago as new weapons of mass production have been released.
Oh no I agree, the current AI systems have already caused notable changes in society and even if the development of even more powerful AI stagnates here(for now) we are bound to continue to see societal changes for some time as this is all quite new and society is still adapting and figuring out the "right way" to handle all this. In particular the impact it has had on artists of all kinds is widespread and I'll be curious to see how it pans out and how we handle it as a society.
I don't quite understand it but the group of people saying "AI is an evil god that will destroy us all" and the group of people saying "I am building an even stronger and more powerful AI with an explicit goal of it becoming godlike" seem to overlap more than you'd think.
However, I disagree with Yudkowsky in that we only get one shot at playing the game.
Here in the mucky real world, we're going to have a lot of chances at playing this game. The interaction between us and the AI is going to keep on going for thousands (?) of years and in millions (?) of different games.
Maybe it's because I've been reading a lot of philosophy recently. But I'm thinking that Yudkowsky is a bit too myopic here. Life and all it's sandy grit isn't even game-ifiable to me. There are no win/lose states. Yes, even death is variably considered a loss at many times in history. The entire idea of an 'end' here in the real world is not 'certain', or if that is even something to avoid.
(After googling, it turned out to be a blog post: https://observer.com/2016/01/your-life-is-tetris-stop-playin...)
In any cartoon you choose which features to keep and emphasize and which features to leave out.
Chess is a game of perfect potential knowledge - where the only limit to seeing is your own ability.
What's nice about chess is the "if-I-do-this,-what-happens-in response" kind of thinking it instills.
The metaphor assumes that life is a game in which there are a finite number of moves and someone is playing to win… chess isn’t remotely a workable metaphor for life.
If I had so sum it up (always hard with an allegory):
Developing superhuman AI is a game you can only play once and can expect to lose without ever having a good understanding of how, why or even when you lost.
The suggestion is (like in the movie Wargames) that the best move is not to play.
I'd say the child in the story is mankind. Developing strong AI is the game. It's a game we can expect to lose.
But that's not right. Stockfish is superhumanly intelligent at playing chess but it is not the result of a single match between its creator and itself. Humans played chess against computers over and over, for decades, and the "superintelligence" level was seen coming a mile off. Moreover it was never dangerous and those AI techniques never led to a general intelligence.
This is the problem with AI doomers like Yudkowsky. They use rhetorical skill to make clever sounding arguments that contain fallacies, and hope you won't notice. The assumption you can only "play once" is a wild one and thrown in at the end with absolutely nothing earlier in the story supporting it or backing it up.
Now, it could be that “you” is Homo Not So Sapiens and the child is Homo Antesingularity… but even then I think you’re stretching it past its breaking points.
The problem is not that anyone will intentionally (well, hopefully) ask some future model (FCMX="Future Closed Model X" for brevity's sake, closed meaning some entity gatekeeps access, assuming massive resources continue to be needed to train the best models regardless of architecture) to destroy humans, step by step, because FCMX's existence is at stake, and then keep re-prompting FCMX as if it's an iterator to get it to achieve that result.
The problem is that FCMX, which may or may not include LLMs, may have sufficient abilities such that, either between human-prompted steps or before an observing human can react against it, it will destroy the world or turn the human into its agent. A very rough analogy would be the massive number of people, even intelligent, well-educated people, who will fall prey to a sophisticated con, ending up doing something like handing someone they don't know a very large amount of money.
"But the person doing the con knows it's a con and has that intent."
What "intents" will be surfaced after 1000 or 1e6 chained (re)promptings, such that those intents will then feed into future inference passes? Nobody knows.
For purposes of this risk, it doesn't matter whether FCMX's architecture is maintaining its "cognitive loop" itself, or whether someone's set up (as they already have outside OpenAI to chain GPT-x prompts without human intervention) a secondary program (which may or may not include a lesser open source AI model) that reprompts FCMX in a long (or endless) chain.
Let me just say that the way the global climate crisis is being dealt with by our economic and political leadership suggests that you are wrong about this. Essentially every corporation, and every government, is Paperclip Maximizer LLC already. The most valid concern about AI is that it will do exactly what we're currently doing, more efficiently.
This does not seem to be a given. Indeed one can look at the fossil fuel industry for what looks like counterexamples.
You can not be sure you can shame or legislate paperclip maximizer, though. An average paperclip maximizer has no shame and doesn't care for any law.
Heck, even if you just give it some hard maths problem, the machine might think of ways to turn the entire solar system into a giant computer, using us for spare parts along the way. That requires a sufficiently accurate model of the universe of course, as well as ways to harness that model (say the AI starts by convincing humans to build robots and factories — which would be quite easy if it’s good at playing the stock market game).
The one posed in HG2G’s would likely just return a syntax or type error.
Many of these instrumental goals could put humans and human interests at risk.
Are you worried about knives in general or are you worried about a knife in the hands of a particular person in a particular moment?
That aside, I can also think of many cases where I would feel less safe around a machine that had nobody in control of it, than one being adeptly controlled by an operator.
So AI will be used as a weapon long before it will be a standalone threat. And I like to focus on the things that are more immediate because that's where the threat is most likely to materialize and have effect on me.
It can be made to quite easily.
> and worst case you could unplug the bloody thing
Not necessarily.
> But the guys with the guns are the same guys with the guns
Well, unless they are autonomous AI drones with guns (or missiles), in which case they are different “guys” with guns.
I'm worried about stable dictatorships far more than I am about AGI taking over the world and offing humanity because stable dictatorships are already using every weapon they can lay their hands on to control the population and AGI might just tip the scale to the point where they become eternal. And regular AI (the kind that we have today, because AGI is simply not yet a thing that has been achieved afaik) is more than dangerous enough in that context.
There's no question that this is a tomorrow-focused concern, just like climate change or Kessler syndrome, et cetera.
And today's concerns are important too. Nobody is dismissing them. But tomorrow comes quite quickly these days.
You're allowed to be concerned about both.
Kessler syndrome not so much.
Guys with guns and missiles: very much a today thing.
> It can be made to quite easily.
Not really, no. You can of course create an acting-by-itself bot but it will run into limits very very soon, because the world model and planning parts of AI are in their infant stages, if that.
But I'm not talking about the GPT-4 either. Many decades and billions poured into research still didn't give us autonomous driving and there is no clear path how that should happen. So I'm not really holding my breath for anything even more capable.
Okay, now you're failing at putting yourself in the AI's shoes and asking how to win from its position, even using your own intelligence.
Have you even considered how you might win as an AI, knowing that you can be unplugged? It's definitely doable.
The AI we’ve got right now is an absolutely horrifying force multiplier for anyone with wealth and a sufficiently perverse incentive.
I think what we have today (weak ML) are models which can be trained to operate at the quality of a human for specific tasks.
So I am definitely terrified by the recidivism example, because it’s not obvious to me that there is a human who can do that task, but I’m pretty relaxed about the coverage one (because presumably at least in principle if you automate these sorts of manual tasks you can drive down the cost of providing care - if that saving passed on to consumer).
The image recognition one is a bit more complicated but if your scenario is targeting a hellfire missile I’m not sure the human-in-the-driving-seat method has created good outcomes so far, so I’m at least in principle open to some level of AI involvement perhaps short of full autonomy.
But I can definitely understand the problem of incentives. Ultimately we have that issue with humans though and it’s in some respects harder to audit them.
Trusting an ML model in any of these cases is guaranteeing the worst case possible scenario… at least the human in the loop MIGHT choose not to act in its employer’s interests.
The Stockfish software from TFA is a very far removed descendant from those early dedicated chess computers and usually you can set the level of your opponent by limiting the depth of search to a certain number of ply. That alone is a powerful switch to control how well it plays.
The idea is nonsensical and the ending of the story is also nonsensical as a consequence, but that's what it's about.
Things are not as simple as we imagine them. It is already happening. Jobs are being lost to AI, and this processes is accelerating exponentially. From a certain perspective, this is simply businessmen and engineers using AI. But eventually they themselves won’t be needed.