Now we're talking! The testing thing is indeed impossible to formulate using the code, it has to be formulated in language. But that is not an issue, because (in most bootstrap scenarios I'm considering) language acquisition happens before (or alongside) acquiring the intelligence necessary to understand itself to the point of re-formulating itself. So we
can formulate the tests in English, and subject to the language acquisition having succeeded (which we can establish independently) we can trust that the tests are correctly understood.
This actually makes the tests more robust, because the AI self-describing its own plans in its own internal "vocabulary" is quite able to recognize vague associations, such as the is-it-death-or-not scenario you describe. If it is a similar enough conception that it can be (internally) described as "death", (past a threshold), it will (with a testably sufficient probability) trigger the associated constraints.
The idea that a "buggy" intelligence could acquire language, and understand it at a level sufficient to "digest" advanced technical material, is sufficiently unlikely.
More to the point, the "ethical" tests and constraints are only the topmost level of a pyramid of testing that ensures that all the pieces perform exactly as intended along the way. The system at no point behaves in a random or inexplicable way. So by the time such capabilities are unlocked we know that the algorithm works in accordance with theory.
You are also completely right that this initial "dumb-almost-AI" would have to be pretty smart already. The trick is that it grows within a set hardcoded framework at first, which is sufficiently flexible to allow it to in fact perform language acquisition and (later) intelligent design.
Only then would re-writing its own hard-coded base become interesting. Therefore the design I am working on already passes human-level capabilities. But that does not mean the AI cannot improve itself even further. And the trouble in the context we were discussing it in is that we need to be sure that if it does rewrite itself - and in a way that we might not be able to easily comprehend - it still cannot break out of the mold.
As far as the goal function is concerned, it is as you can imagine rather complicated to do. And the solution will undoubtedly differ based on the architecture you are working in.
What I'm doing personally is pretty straight-forward; it involves iterative constraint solving using quantitative metrics to increase trust and/or doubt in various concepts and plans, and this trust is established based on consistency and coherence with previously acquired (probabilistic) background "knowledge" - as well as consistency within any single context of thought. A single context might be a sentence or a paragraph reading, or a logical puzzle being solved, etc.
Over time the probabilities increase (at least, in theory), both for any single consideration and across the entire knowledgebase. (See NELL [1] for comparison. Except, it's a lot smarter than NELL.)
So the goal function is a fairly generic optimisation algorithm, and the initial goals defined are equally generic - increasing overall consistency, coherence, and efficiency. (Measuring these is fairly straight-forward.)
The trick is in setting up the reward structure autonomously, so it keeps advancing itself. The learning algorithm needs to create sensible (long-lived) intermediate goals and maintain their relative priorities over time. Among other things.
(I really can't explain how it works though without going over a whole bunch of other supporting algorithms).
In any case, to begin with the A.I. need not have a desire to do anything at all except learn - and it improves within the constraints of the hard-coded base framework. But the model can still advance (at least, in theory) to the point where it acquires the ability to comprehend its own design and re-engineer itself completely.
By this time however it is able to understand and apply the ethical constraints we defined in English, alongside all the other intermediate goals (but with specific higher priority).
[1]: http://rtw.ml.cmu.edu/rtw/