Hacker News Reader

You might be partially correct but you seem to omit built-in wants e.g. hunger or thirst or lust.

Furthermore, if a machine has no wants/needs then it won't take any action at all. It will just stand there and won't even experiment to find the good or bad outcomes.

View on HN

If not doing anything has bad consequences, then it will do something to avoid bad consequences. It seems to me that "want" and "avoid bad consequences" are quite isomorphic

It would need "to want" "to avoid bad consequences". It is rather circular because of the word "bad" which implies "do not want".

If it is indifferent to all consequences, you are back to square 1, so no, consequences are not the missing magic. Consequences are a function that transform one "want" into a different "want".

You aren't back to square one. Even indifference will result in selection so long as the indifference is demonstrated in different ways by different AI. The ones which have behavior which accidentally maps slightly better to good outcomes will so better. So long as that gets propagated more strongly to the next generation of AIs than average, you get selection for good outcomes without any explicit 'want' function

Perhaps, while slowly learning, the machine will cry.

The machine is me.

Reply on news.ycombinator.com