Hacker News Reader

You've read too much bad science fiction.

I am not worried about AGI. We're nowhere near it. We don't even have a good idea what it would look like.

I am worried about people succumbing to hype or greed and using badly understood algorithms to control critical infrastructure. We already have a preview of what that can look like with Facebook's and Google's content filtering and recommendation algorithms aiming for ever higher "engagement". It's not pretty. Other examples include HFT bots and Amazon's pricing bots. It's funny to see $10,000 book on sale. It's less funny to see a flash stock market crash. It will be totally not funny if something like that will create a global economic crisis through some subtle yet "wide" feedback loop no one is aware of.

It is impossible to justify saying with absolute certainty that AGI is nowhere near. You don’t know that. And in this case you have to assume the worst case, not the best case.

AGI will come when the substrate for AGI is laid down. We probably have already done that. As cloud computing matures, we will approach a world where every computer offers its computational resources on a global compute market. At some point between here and there, we will reach a place where compute is cheap enough that experiments will occur regularly that are sufficiently massive to dredge up the solution. And improvements in MRI fidelity, underlying improvements in computing technology and other things will only shorten the fuse. There is no reason why this couldn’t happen tomorrow.

Only one thing is sure: without a computational substrate to stir from, AGI cannot come to be.

Nobody is going for absolute certainty here. That bar is too high in any conversation.

His point was mostly that, way before you achieve the kind of AGI portrayed in fiction, you'll have semi-intelligent interdependent systems that cause a lot of trouble due (like the kind that already happens to a lesser degree). Those are the ones that we should worry about right now.

That is not even obviously true. And even if it were it wouldn’t make sense. You’re going to wait until the tremors to get ready for the earthquake?

"I am worried about people succumbing to hype or greed and using badly understood algorithms to control critical infrastructure."

What you're describing as hype/greed around badly understood algorithms could easily manifest as 'cult-like oracle religions'.

Combine that sort of thing with the superstitious, wealthy people playing power games, and what you're describing is essentially the exact same thing as the "bad science fiction" described above, just from an arguably more grounded angle.

Except - We're nowhere near it.

Using Siri, not itself the outmost edge, but an estimate of the state of the art... If you have an iPhone, take it out and ask Siri how much memory is left on your phone. If she doesn't know how (she doesn't) explain it to her (she can't learn). This is not a philosophical question, it's not a question that requires opinion forming, or fuzzy guesses, it's not a question that requires establishing ad hoc priors that update in real time, it doesn't need to parse an idiom, or draw connections, or to disambiguate complex meanings. It's a simple request, and a simple task. To anyone overly paranoid about the AI issue, this might actually help attenuate those fears when someone starts going on about the singularity being upon us... you just need to take out your phone and ask Siri how much free memory you have left; if she tells you... panic.

An iOS dev is going to read this and build that feature into Siri just to mess with you.

I hope so hah. It would be nice if Apple made it so, instead of Siri saying "sorry i cant help you with that" she said "i dont know how to do that, can you teach me". And even if it were not really AI but macro-based, it would be better than nothing.

>We're nowhere near it.

So?

>We don't even have a good idea what it would look like.

This should make you even more worried.

Aren't FLOPS/dollar still increasing exponentially? "Nowhere near" means something different when you are dealing with exponentials.

If that growth doesn't cap out before AGI, then at some point we'll harness more computational power than a human brain, in a more controllable way. And around that point AGI probably goes from impossible to possible to easy in a stretch of about 6 years.

edit turns out 6 years was about the timespan for computer Go to span "plays like a competent amateur" to "superhuman". There was barely anything worth reporting before 2010 [0].

[0] https://en.wikipedia.org/wiki/Computer_Go#21st_century

> It angers me that Peter Thiel simultaneously advocates AGI and maintains a hardened bunker.

Yes, because if there's one thing that will stop a paperclip maximizer, it's a bunker made of useful matter...

I'm far more worried about the long-term consequences of raging inequality and climate change myself. I suspect I'm going to see both of those in my lifetime.

I'm not so confident that I'll see an AGI in my lifetime, but even if I do, I'm not going to assume it has a hindbrain that drives it to maximize all the paper clips at our expense.

Look, it is inevitable. How are you going to stop it? Forget AGI, even regular AI is too sweet to not develop.

This is a reference to Oppenheimer:

“However, it is my judgment in these things that when you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only after you have had your technical success. That is the way it was with the atomic bomb. I do not think anybody opposed making it; there were some debates about what to do with it after it was made. I cannot very well imagine if we had known in late 1949 what we got to know by early 1951 that the tone of our report would have been the same.”

This is one of very few things that once created, inherently doom the world. It is possible for humanity to come back from nuclear war. There is zero possibility of surviving AGI proliferation. Therefore, the option of never developing it in the first place demands consideration. It doesn’t hurt you to just consider it.

Cloud computing will soon make compute-as-service cheaper than anyone ever imagined. It will be much cheaper to use cloud computing than to have your own super computer. It’s only logical to imagine that the huge, high-compute experiments that crack AGI will be performed with cloud computing. Shutting down the internet is very feasible and would significantly increase the cost and difficulty of performing experiments. Large super computers are all owned by large and well known corporate and academic entities and therefore shutting down all supercomputers through regulation is feasible. With these two coarse measures, we would buy ourselves enough time to implement more broad and subtle solutions. Yes, this seems insane but we are confronting certain existential doom. It would be worth it to at least try.

That’s not enough. Google trained their AI internally. They don’t need the Internet. You’d have to go after every organization in the world, forever.

What do you really care, ask yourself will you recognize humans in 1000 years? Why do you prefer there be one kind vs another kind of far more intelligent thing in 1000 years? If atheism is true then who really cares?

https://m.youtube.com/watch?v=5U1-OmAICpU

> AGI is the single biggest existential threat on the horizon

> There is zero possibility of surviving AGI proliferation

You're spreading unsubstantiated FUD. In another post from your account you suggest it's unethical to have children because AGI will be so bad.

The vast majority of experts do not support any of these beliefs. Most experts believe we are not anywhere near close to AGI, and/or that we are missing fundamental components required to create it. Even when/if we do create it, most organizations recognize AI safety and policy as an important area that is actively worked on already.

If you want to be concerned about AI, be concerned about military weapons technology, unethical profiling and tracking, or methods for invading privacy. These are concerns that actually have a basis in real technology.

The condescending tone you have does not help you. And you seem to be more concerned with looking at my account history or brushing me off as “fud” than actually addressing the substance of my comment. And yes, I do so completely believe in what I’ve written that I have opted to not have children until it is clear that a solution will be implemented.

There are two parts to my argument: the substrate for AGI to spring from is basically here or around the corner. Human level ai is fundamentally incompatible with human society. Pick the one that you think is wrong and tell me the chain of logic that proves I’m wrong.

It is obvious to anyone that AI has the potential to be more dangerous than anything humans have ever encountered or created. We can come back from viruses and nuclear bombs and colliding with interstellar columns of gamma radiation. AGI is the first thing ever to pose the threat of truly wiping out humanity. And then there is the question of how the economics would play out of it didn’t destroy us, which I have shown informally to not work out very well. At the bare minimum, this demands caution and proactive, defensive measures. The burden is on YOU to prove it’s safe. So please save your “fud” and anything else that does not address the core of the matter.

I happen to agree the burden is on people to prove it’s safe. But that’s not how things work. Take Climate Change for example. How many people take the uncertainty as a license to keep going off the cliff?

Read this story as a possible alernative:

http://m.nautil.us/issue/53/monsters/the-last-invention-of-m...

There was no substance to your comment to address. You can't make a bunch of unsubstantiated claims and then demand that you be proved wrong or else require that everyone default to assuming you're right. You made a claim, the burden is on you to support it.

You're making many scary claims about AI with no evidence to support your theses, along with a unrelated reference to natural selection. Both of the key arguments that you pointed out, 1) "the substrate for AGI to spring from is basically here or around the corner", and 2) "Human level ai is fundamentally incompatible with human society" were presented without any evidence and are unsupported by nearly all major experts in the field. FUD was not a distraction from your argument but a title for what it is: Fear, uncertainty, and doubt.

But, in case you want to read up, here are some experts contradicting your first claim:

1: https://www.technologyreview.com/s/608911/is-ai-riding-a-one...

2: https://www.quantamagazine.org/machine-learning-confronts-th...

3: https://www.axios.com/artificial-intelligence-pioneer-says-w...

And here are some experts contradicting your second claim:

1: https://80000hours.org/problem-profiles/positively-shaping-a...

2: https://medium.com/@deepmindsafetyresearch/building-safe-art...

3: https://openai.com/blog/ai-safety-needs-social-scientists/

4: https://www.scientificamerican.com/article/artificial-intell...

5: https://www.technologyreview.com/s/602410/no-the-experts-don...

> AGI is the single biggest existential threat on the horizon

Climate change says hello. (I suppose you could argue its here already)

Yes. Natural selection does not end when biology does. Anything that has scarcity and variation will have selection effects: https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

And should AI go wrong, Theil's bunker will not save him.

Great link, thanks for sharing. That Alan Ginsberg poem is heavy duty. It reminds me a lot of this quote from Revelation (which I read as an epic proto sci fi):

> The second beast was given power to give breath to the image of the first beast, so that the image could speak and cause all who refused to worship the image to be killed.

Thank you so much for posting that link. I am on the verge of tears. I am tortured day and night by my conviction on this issue and reading that link made me feel for the first time that I am not alone in this conviction. This man understands completely where I’m coming from. I had no idea there was a formal philosophical name for what I was trying to describe.

Every day I experience the world and I am overwhelmed by how amazingly good it is. I relish every moment and everything is rich with opportunity. But as your link shows, the world is only this way because human society and everything in it happens to be the natural conclusion of all the worlds entities racing to the bottom. We live in a magical time where empathy and cooperation are often the most expedient thing to do. All that needs to happen is for the economic equation to change a little bit and the world will contort into something else, maybe something grotesque.

I think it is also based on how you perceive the world and how you treat the forces that you don't have control over. Human empathy and tribalism arose from the benefit of working together against natural selection, and there's no guarantee for AGI to 'go rogue' when they could also just as easily recognize the importance of working with it's creators and peers. Again, the stuff you're afraid of is mostly the dystopian scifi at this point, meant to generate a good story more than accurately predict future states, so don't sweat it.

Your comment betrays your total lack of understanding. Yes, one version of the AI could exist just as readily as another. The point is that not just any random version can persist. Your logic basically rejects the concept of natural selection. I say it as a friend: you have not even scratched the surface. You have a lot of thinking to do.

You think that I’ve watched too much sci-fi because you can’t see what I see. Sci-fi treats AI in a very friendly way. Halo and Star Wars are filled with AI agents and those depictions of AI totally ignore several huge economic inevitabilities that come with the presence of AI. AI is vastly more likely to be depicted as cute side-kicks in sci-fi than anything else. And obviously I don’t believe what I believe because I watched the terminator. I used to be totally pro-AI singularity guy. I’ve been contemplating these issues for a decade and have only now started to be accused of watching too much sci-fi. Just think it through carefully and I guarantee you will at least partially agree with me.

I disagree, and nothing typed in your condescending dismissal supports what you're saying - either about me, natural selection, or AI in general. It seems like you're projecting some deep seated paranoia onto AGI, and judging from your defensiveness, I would recommend talking to someone professionally about it. Again, you're sweating it a bit too much for what's healthy ¯\_(ツ)_/¯

I agree with your comment, but just a minor point: has Thiel made it clear that his bunker is to protect against potential AGI threats? I thought it's insurance for nuclear warfare.

You’re right. It was just a kind of loose association and it’s not related to or important for the core point of my comment.

Just as a subjective thing, the bunker doesn’t strike me well. If peter is advocating anything that has the potential to upset the global economy/global order then it should be him to stick his hand in the proverbial hole first; but apparently he’s going to let us do it while he watches from the safety of his bunker.

A bunker might help with the initial destabilization caused by AI but it would of course not help him if one of the more horrible outcomes were to transpire.

AGI is almost certainly not the single biggest existential threat. And I don't really get why people think it might be.

> AGI is the single biggest existential threat on the horizon.

Really? The madman with the football in a white house and the media pushing an anti Russia sentiment aren't scaring you at all?

Nuclear war is not be capable of wiping out humanity (although causing potentially billions in casualties), a hostile AGI can likely wipe out humanity to the last person.

It can? How exactly? I can never wrap my head around this argument from the AGI risk people. I simply cannot in any way follow their logic. Why is this imaginary creature maniacally homicidal? Please give me a good reason for this.

The only correlation I have noticed between intelligence and violence is generally negative (sure this is completely unscientific). Like my friend, who is a pure mathematician, and won't eat animals because it deeply troubles him to harm other animals. Haven't you noticed that smarter people tend to be pacifists? Look at animals like lions....fabulously homicidal. Again, this is a crap analogy, but I'm just making the point that there are other ways of looking at the intelligence/homicide correlation.

Secondly, in all this stupid talk of homicidal AGI's I never hear one good mechanism proposed for how the imaginary creature is going to execute all 7 billion or so of us, it's plan for disposing of our bodies....etc. Oh wait I forgot, it's smarter than us...so of course we wouldn't know...we can't imagine it with our feeble brains. Not to mention that it creates a fabulously efficient source of eternal energy for itself, lives in cyberspace in some magical bit realm--and then here comes the million dollar question: Why the fuck would it spend the energy to execute all of us if it can just ignore us? Sound a lot like god delusion to you?

I really can't stand when people trivialize nuclear war like this. 'It can't kill all of us'. Please just shut up. Will it crumble the fabric of our society? Leading possibly (remember, all AGI people care only about possibility--not even likely realities) to war and famine? Who cares if not ever single human dies, what is the consequence to our world fabric? Let me save you the suspense--it's completely devastating. These arguments coming out of CFAR and the effective altruism movement are so fucking dumb I constantly want to scream.

You’re completely wrong. There are so many corrections I would need to make that I can’t even write them here. If I could talk to you in person or on the phone then I might stand a chance of conveying it all. Is there some way I could PM you my contact info? I’m going to put my email in my bio so you can reach me there.

Yes, I am so sure and so serious that I would be willing to take this to a phone call or meetup. Just to change one persons mind.

What is your background? I've been studying math, neuroscience , AI, biology etc for a long time. I left quantitative finance (after leaving a math program) to work at a university developing computational biology software. In finance I worked with neural networks well over a decade ago before people in Silicon Valley had even heard of them. Convince me you're not a kook and I'd be glad to have a conversation.

The problem with AGI is not that it wants to run a human-like society sans humans (I don't think AGI wants war in a human sense), the problem is that it may have goals (like famous paperclip maximization) that are incompatible with human continued survival. Humans don't need to be "executed and disposed", they could simply be lost to habitat loss (eg oxygen rich atmosphere can be a pesky thing, why not strip it? or, temperature regulation needs, or whatever. It doesn't take much global terraforming to wipe out humanity inadvertently).

Reproducing AI wants energy. We want energy. Somebody is going to win, and it’s not us.