It's akin to trying to come up with a firearm that can't kill innocent people. The problem isn't even coherent.
The only solutions are:
1: Redefine what AGI means, like openai has done.
2: Prevent AGI altogether
For instance, in your firearm example, we can't determine who is innocent and who isn't. But we can use embed a facial recognition device into the firearm that then cross references whoever the firearm is pointed at against a database of known non-innocents.
And then exclude any targets obviously under 18.
And then make the model probabilistic and add greater weight to targets that appear to be carrying weapons of their own or that are acting in an obviously dangerous manner.
You get the point. After enough time, you would have a firearm that is (arguably) better off than the one you started off with and so closely approximates the goal of your initial problem description that you don't care to make a distinction.
I could be wrong, but I imagine this is how AGI safety researchers think about their work to some degree.
How sure do we need to be that we're safe from evil AGI? Once it's in the wild, we don't get another chance to make it safe.
Once evil AGI arrives, that's quite possibly binary. Our only hope is good AGI outsmarting it and at that point we're playing the same game that produced us. We're not universally good. So we have to be more sure of it then anything, so we don't go down that road.
On this issue, I'd prefer the truly paranoid are in charge of safety. Not the gung-ho "of course it'll be nicer then us" crowd.
I don't think we can constrain it. Our best hope is to discover that it wouldn't care to harm us.
That explicit reasoning approach falls squarely into GOFAI, symbolic reasoning, expert systems etc... so we're well past that at this point and know the problems with it.
Anyway, again, it's not a tractable problem. Something smarter than you is going to be able to eventually beat whatever restrictions you put on it. Might as well just be comfortable with it.
The symbolic reasoning you're talking about is going to make a comeback through probabilistic inductive programming, so don't dismiss it. It is necessary to attain sample efficiency and generalizability, which is virtually impossible with purely statistical approaches. I recommend you look into Josh Tenenbaum's work: http://science.sciencemag.org/content/331/6022/1279.
Rather, the AI community has really good confidence that you can "fool" every AI approach to date. That has been true the longest for GOFAI approaches, and now we're doing it for DL/RL.