Hacker News Reader

Comment by duvenaud | Hacker News Reader

This is a great point. But I'd say that capable entities have a habit of turning themselves into agents. A great example is totalitarian governments. Even if every single citizen hates the regime, they're still forced to support it.

You could similarly ask: Why would we ever build a government or institution that cared more about its own self-preservation than its original mission? The answer is: Natural selection favors the self-interested, even if they don't have genes.

View on HN

Now agency is an end unto itself I wholeheartedly agree.

I feel though, that any worry about the agency of supercapable computer systems is premature until we see even the tiniest— and I mean really anything at all— sign of their agency. Heck, even agency _in theory_ would suffice, and yet: nada.

I'm confused. You agree that we're surround by naturally-arising, self-organizing agents, both biological and institutional. People are constantly experimenting with agentic AIs of all kinds. There are tons of theoretical characterizations of agency and how it's a stable equilibrium. I'm not sure what you're hoping for if none of these are reasons to even worry.

None we have made from unliving things no. «agentic» is a 5$ word of ill construction. The literature is littered with the corpses of failed definitions of «agency», «planning», «goal dorected behavior». ‘Twas the death of the Expert System AI (now it’s just constraint solvers). It will be the death of attention/transformer AI before long, I wonder what banality we will bestow upon it.

Okay, well it seems funny to both claim that there is no good definition of agency, but also be sure that we've never demonstrated any hint of it in theory or practice. Planners, optimizers, and RL agents seem agentic to some degree to me, even if our current ones aren't very competent.

You know, i thought of the right response just now, browsing my flamewars two weeks later.

«Some degree» of agency is not even near sufficient identification of agency to synthesize it ex nihilo. There is no Axiom of Choice in real life, proof of existence is not proof of construction.

Reply on news.ycombinator.com