Computerphile has a series of videos on AGI that touches on this topic, using the lens of "define a human to a computer," a seemingly simple task with a huge slew of ramifications that come with it. Basically exploring whether or not Asimov's Laws are actually useful or practical (they aren't, and this is a huge reason why).

The short version of the specific video in the series about this explores how edge cases become extremely relevant in this domain, and asks whether or not an AGI would consider these to be people or sufficiently people-like:

* Dead people

* People in vegetative states

* Unborn people

* Simulated human brains

* Sufficiently intelligent animals

And so on. We can't simply list every edge case, that list would be literally endless and the developer is guaranteed to miss one that the AGI will eventually find.

They all come with implications. How can an AGI never harm a human without also accurately predicting extremely discrete details from the future? Any action it takes may cause harm or death to one or two humans 50 years later, and it can't really know for certain. It can probably known if its actions will affect hundreds or thousands, but not one or two.

Suddenly a whole lot of deep moral judgements have to be made, some of them nearly impossible for a human (e.g. the aforementioned unborn human example above). The Three Laws of Robotics unfortunately rely way too much on human intuition, something that is extremely difficult to model in a machine.

I may be remembering incorrectly, but the whole point of that story was how those laws fail spectacularly. That's a critical detail. But I digress.

Clearly an AI designer isn't qualified to just make those judgements on their own, and probably doesn't want to anyway. Really no one, either living, dead, or unborn, is qualified to be the judge and jury in this case, and a committee isn't necessarily better.

It's a fascinating topic, to be sure, but I'll admit that it's not my area of expertise.