Hacker News Reader

Comment by etienne618 | Hacker News Reader

I hear you - its just that I dont think that it neccesarily follows that an AGI in whatever form will necesarily be unconstrained or free to improve itself. I also happen to think that the biological systems are in-fact rather special and hard to replicate - especially in terms of efficiency and resillience. Just because some implementation of a universal turing machine can simulate intelligence doesn't mean it can do it well enough to survive the real world.

Finally, to me, nuclear reactions are kind of the opposite of AGI: I think it's vastly easier to blow something up (increase entropy) than to create an intelligence capable of understanding and improving itself (decreasing entropy - possibly at an accelerating rate).

View on HN

Yeah - it doesn't necessarily follow, but the behavior of people working on the technology doesn't inspire a lot of confidence.

Even if people were trying to constrain its access seriously I think that's unlikely to work (hard to contain a superintelligence that wants to not be contained - it's possible to trick a chimp to go into a room and the delta in intelligence between a human and a superintelligence is way bigger than us and chimps).

Instead I mostly observe people not really understanding the e-risk argument, focused mostly on small stuff that doesn't matter as much (AI language, bias). The people developing the tech connecting it to the internet and expanding capabilities, giving it access to code/training ability to write code, preparing massive datacenters for it, etc.

All of this without really understanding how to align it or what it's actual internal goals really are.

> "Just because some implementation of a universal turing machine can simulate intelligence doesn't mean it can do it well enough to survive the real world."

This could be true, but I would bet against it - and the downside risk of being wrong (potentially complete extinction) means it seems worth being way more cautious about it than we (humanity broadly) are observed being.

Nobody says that it necessarily follows, but it's a quite trivial step once you have the AGI, hence pointless that you bring up it as an argument.

While I appreciate you find my argument persuasive - imo this style of comment will do more to alienate those replying to me that disagree than to persuade them.

They're likely irrelevant anyway. We're all shouting into the void.

Reply on news.ycombinator.com