Hacker News Reader

Comment by gambler | Hacker News Reader

You've read too much bad science fiction.

I am not worried about AGI. We're nowhere near it. We don't even have a good idea what it would look like.

I am worried about people succumbing to hype or greed and using badly understood algorithms to control critical infrastructure. We already have a preview of what that can look like with Facebook's and Google's content filtering and recommendation algorithms aiming for ever higher "engagement". It's not pretty. Other examples include HFT bots and Amazon's pricing bots. It's funny to see $10,000 book on sale. It's less funny to see a flash stock market crash. It will be totally not funny if something like that will create a global economic crisis through some subtle yet "wide" feedback loop no one is aware of.

View on HN

It is impossible to justify saying with absolute certainty that AGI is nowhere near. You don’t know that. And in this case you have to assume the worst case, not the best case.

AGI will come when the substrate for AGI is laid down. We probably have already done that. As cloud computing matures, we will approach a world where every computer offers its computational resources on a global compute market. At some point between here and there, we will reach a place where compute is cheap enough that experiments will occur regularly that are sufficiently massive to dredge up the solution. And improvements in MRI fidelity, underlying improvements in computing technology and other things will only shorten the fuse. There is no reason why this couldn’t happen tomorrow.

Only one thing is sure: without a computational substrate to stir from, AGI cannot come to be.

Nobody is going for absolute certainty here. That bar is too high in any conversation.

His point was mostly that, way before you achieve the kind of AGI portrayed in fiction, you'll have semi-intelligent interdependent systems that cause a lot of trouble due (like the kind that already happens to a lesser degree). Those are the ones that we should worry about right now.

That is not even obviously true. And even if it were it wouldn’t make sense. You’re going to wait until the tremors to get ready for the earthquake?

Aren't FLOPS/dollar still increasing exponentially? "Nowhere near" means something different when you are dealing with exponentials.

If that growth doesn't cap out before AGI, then at some point we'll harness more computational power than a human brain, in a more controllable way. And around that point AGI probably goes from impossible to possible to easy in a stretch of about 6 years.

edit turns out 6 years was about the timespan for computer Go to span "plays like a competent amateur" to "superhuman". There was barely anything worth reporting before 2010 [0].

[0] https://en.wikipedia.org/wiki/Computer_Go#21st_century

"I am worried about people succumbing to hype or greed and using badly understood algorithms to control critical infrastructure."

What you're describing as hype/greed around badly understood algorithms could easily manifest as 'cult-like oracle religions'.

Combine that sort of thing with the superstitious, wealthy people playing power games, and what you're describing is essentially the exact same thing as the "bad science fiction" described above, just from an arguably more grounded angle.

Except - We're nowhere near it.

Using Siri, not itself the outmost edge, but an estimate of the state of the art... If you have an iPhone, take it out and ask Siri how much memory is left on your phone. If she doesn't know how (she doesn't) explain it to her (she can't learn). This is not a philosophical question, it's not a question that requires opinion forming, or fuzzy guesses, it's not a question that requires establishing ad hoc priors that update in real time, it doesn't need to parse an idiom, or draw connections, or to disambiguate complex meanings. It's a simple request, and a simple task. To anyone overly paranoid about the AI issue, this might actually help attenuate those fears when someone starts going on about the singularity being upon us... you just need to take out your phone and ask Siri how much free memory you have left; if she tells you... panic.

An iOS dev is going to read this and build that feature into Siri just to mess with you.

I hope so hah. It would be nice if Apple made it so, instead of Siri saying "sorry i cant help you with that" she said "i dont know how to do that, can you teach me". And even if it were not really AI but macro-based, it would be better than nothing.

>We're nowhere near it.

So?

>We don't even have a good idea what it would look like.

This should make you even more worried.

Reply on news.ycombinator.com