Comment by ggreer | Hacker News Reader

Comment by ggreer | Hacker News Reader

Parent Full thread

Most experts don't think this is likely, just that the potential consequences are bad enough that it's worth making it less likely. It's just like how most people don't think global nuclear war is likely, but it's still worth reducing the likelihood of.

Also, the people who worry about this aren't concerned about current ML stuff going haywire. They're worried that we're one or two algorithmic breakthroughs from something that can improve itself. If the upper bound for what sort of intelligence is possible is much higher than us, we could quickly be outclassed. As Nick Bostrom says:

> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.

If you want to delve into the best version of the AI risk argument, I recommend Superintelligence by Nick Bostrom.

There are actually no experts on AGI. Just opportunists trying to sell their books. It's impossible to be an expert on something that doesn't exist. It's like being an "expert" on Sasquatch or the Loch Ness Monster. A total joke.

If we had billions of dollars being spent on genetic research towards build giant Scottish aquatic lizards, than a Loch Ness expert would be valuable.

The lack of imagination on display here is astounding. Billions are being spent by DeepMind and OpenAI explicitly targeted at AGI.

Billions are being spent by DeepMind and OpenAI explicitly targeted at AGI. Not at all... You don't seems to know a lot about AGI. Google, openAI, Microsoft, Facebook, Baidu etc, do NOT fund AGI. They don't fund AGI nor have even a sketch of a roadmap. They fund narrow AI, that is: they fund researchs on specific problems not on an architecture for general intelligence. Moreover, they only fund statistical AI, not causal AI, which is a pathetic local minimum. What you are looking for is opencog or Cyc.

That displays a common misconception exploited by marketers - especially of the Military Industrial Political complex - that what you pay means you will neccessarily get as results.

If you were to try that approach what you would get would be massive boondoggles which gets you via brute force a whole lot of genetic disease afflicted above average size lizards. It is literally the same process as the literal cargo cults thinking that they can just throw resources at an end goal to get a defined result from a defined path without the theoretical understanding backing it.

They're... numbers...

So you're telling me we could come across a series of numbers, that would somehow imbue the computer storing these numbers into sentience. And having these numbers in such a sequence could pose a existential threat to humans.

The people worried about this are having an entirely different conversation devoid of what practitioners call AI.

I've written a blog post on my thoughts of Bolstrom and other AI alarmists:

https://medium.com/ml-everything/ai-optimists-vs-pessimists-...

That's not what I'm saying at all and I've never heard an AI risk person say something like that.

I just read your blog post. It is attacking a weakman[1]. Kurzweil is not worried about AI risk, and his arguments about AI are rather sloppy. Again, most of the people worried about this aren't certain that it will happen. They just think it's plausible enough (say... a 1% chance) that it's worth studying more and reducing the likelihood of.

1. https://slatestarcodex.com/2014/05/12/weak-men-are-superweap...

Studying? From all those years of studying, what has been learnt? I feel like this is deeply stagnant.

I also consider these posts somewhat alarmist, but just because it is represented as a list of numbers doesn't mean it can't be a serious threat to humanity. Emerging properties is a thing - I mean if you put it that way, insert any tyrant here was just a bunch of atoms, or just a particular organisation of cells, yet they were this and that gruesome.

Why do you think that in our insanely interconnected world, where basically everything is either based on or controlled by informatics (numbers if you like) that a program could not pose a threat?

Drones are programmed by numbers and take action in the real world. Some ML-based AI analyze sets of data and perform actions in the real world.

Brains do what? Read signals in different strengths and patterns to take actions in the world.

I'm confused by you thinking there's some great impossibility of code achieving general intelligence. Maybe you're thinking of sentience, or consciousness and view that as an impossibility for AI. I say - an AI can perform actions in the real world as if it were generally intelligent, and whether there's sentience behind really doesn't matter so much from the point of view of humans.

You don't need sentience to be a threat. Just intelligence.

Imagine a super intelligent machine without sentience that just answers questions. It proposes experiments. Takes the results. Tells you things about the world. Gives you advanced technology, robots, fusion, nano tech, whatever.

Now imagine that the super intelligence was owned and operated by someone you don't like. Authoritarians in China, Silicon Valley oligarchs, whoever. The only sentience involved is human, no rebellion, no terminators, and still not a very good result.

You've described google or probably more askjeeves's eternal promise never realized.

Never realized? You very much underestimate the affect google has had on the world. Imagine how different the world would look if Google and other search engines required a 10k / month subscription to gain access.

Yes, they're numbers/weights. Put a lot of them together in an interconnected graph, and let them adjust themselves to improve their performance on a scoring function, and you can get surprisingly complex behavior. Surprising, at least, if you think of it as "just numbers".

Scale that up a few orders of magnitude, and we'll be able to get even more surprisingly complex behavior.

Scale that up some more, and make that scaled up graph into a reinforcement learning system that rewards human-like drives like learning novel tasks and mastering existing tasks, perhaps with a curriculum that ramps up the difficulty of tasks slowly, and allow it to interact with its environment to try things out, and I think you're much closer to developing an artificial version of a young child than you'd think.

And if you asked most researchers what they thought about the course AI research would take 10 years ago, you would get very different answers than you'd get today.

Basically, I think your confidence is a bit unwarranted. I agree that we're more than a decade away, but I think we have a lot of the basic building blocks now.

>So you're telling me we could come across a series of numbers

This is badly missing the forest for the trees. They are not "numbers", they are magnitudes, proportions, relationships, etc, represented as numbers. But these "magnitudes, proportions, relationships, etc" are the stuff that thoughts and behaviors are made of.

And yes, there is a set of "magnitudes, proportions, relationships, etc" in the space of all such possible structures that pose an existential threat to humans. The question is how likely are we to stumble upon it either by accident or intentionally. But this is just a computational problem.

Reply on news.ycombinator.com