> No hardcoded behaviours, no reward functions. - they could evolve in any direction.
If they can hack their reward functions won't this always converge on some kind of agentic opium den?
If they can hack their reward functions won't this always converge on some kind of agentic opium den?
they're only living/evolving to survive, and fork (reproduce).
can't wirehead natural selection if the brain does nothing useful, they'd die and their genome would die with them.