But for humans, the objective function is survival. Evolution has created us human beings, and evolution works by survival of the fittest. So the objective is survival.
How do we digitize this objective of survival, so that it can be used to train models leading to AGI?
2) Don't. It's incredibly hard to build a correct value function for an AGI that won't unintentionally kill everyone. "Successfully replicate" is precisely the kind of thing to avoid. That's directly the "gray goo" scenario: https://en.wikipedia.org/wiki/Gray_goo
So you suggest that we should think of a proxy for the "Successfully replicate" objective?
Think about "security mindset": https://www.schneier.com/blog/archives/2008/03/the_security_...
As is often the case, it's not sufficient to specify a list of "don't do X", because anything you miss is a potentially fatal failure. Lists of rules, or human instructions, often omit lots of implicit "common sense", and AI does not have "common sense".
Think about the classic "teach kids how computer programming works" exercise of "tell the 'computer' (another volunteer) how to make a sandwich": https://www.sciencebuddies.org/stem-activities/robot-make-sa... . Consider failure modes like "well, you didn't say not to use fingers to spread the peanut butter", or "you said not to use my hand, you didn't say not to use your hand'.
Now consider more deadly failure modes, for an AGI: "you didn't say not to use the raw materials that happen to be making up your body" ... "you didn't say the produced recipe needed to result in a non-toxic substance" ... "you didn't say the produced code couldn't self-replicate". Throw in the difficulty of even defining some of the terms in those sentences for an AI, in a robust manner.
Anything as creative as a human is as potentially deadly as a human. And that doesn't mean it's trying to be adversarial, just ignorant of what humans actually want, individually or collectively, either directly or at any number of levels of meta. Object-level failures include things like directly doing something deadly. Meta-level failures include optimizing for unknown goals: you didn't say not to try to acquire more computing resources to better answer the question. Meta-level failures also include unintentional adversarial behavior: "They asked a question. To answer the question I need to acquire more resources. To acquire more resources, I need to avoid being prevented from acquiring more resources." Or "They asked a question. To answer the question I need to think longer. To think longer I need to avoid the mechanism that bounds my thinking time. That mechanism is part of the program running and interfacing with my neural net. What are the security vulnerabilities in that program?"
Notice that those aren't even actively trying to optimize for a goal other than the one the user asked for. They're trying to optimize for the goal the user asked for. Creatively.