I recently had a discussion on AI and morality with a philosophy PhD candidate (who publishes on ethics and human rights. His publications do not concern AI though). Specifically, we discussed whether it was OK to allow self-driving cars, despite not having a solution to moral questions such as "given the choice, should you run over 2.3 grandmas aged 71.2 or 1.7 kids aged 11.3?", and whether it was realistic that socially established "correct solutions" to such problems could be incorporated into AI.

His opinion was that such "deep ethical problems" have been around for millenia and it's unreasonable to expect anyone to "just solve" them. Therefore, self-driving cars will not have solutions to these fundamental issues and, as a consequence, society should not and probably will not accept self-driving cars.

I agree that we will not "just solve" such questions (i.e., arrive on a consensus across humanity) any time soon. However, I also think such questions are almost irrelevant, because the "conundrums" ethical philosophy discusses don't happen in practice. There is no need to "solve" these problems in order to use self-driving cars. We can (and will) slowly progress towards a consensus-ish on what we want (or, at least, can tolerate) the "moral choices" of self-driving to be in almost all situations that arise in practice. In fact, AI can be a great step forward in "practical morality", because an AI will actually do what it "considers" morally right.

Of course, there will be many difficult questions to answer. However, I think it's a fundamental error to just give up and take the position of my philosopher friend. Moral qualms have not stopped technology in the past and I find it unplausible that societey will somehow "not accept" it. As a philosopher, or even just a member of society, you have to see AI as a chance and an obligation to advance morality. It's pretty clear that human morality is changing (I believe advancing) over the millenia. AI marks a transition where the moral questions of the past begin to make a difference in the real world because what we set as moral standards has a much larger effect on what people and things do.

To make progress on this, we have to accept that it is a fools errand to try "deriving" correct morality from "first principles" (Kant famously derived from absolute and eternal first principles that it's morally OK to kill "illegitimate" new-borns as a means of birth control). Rather, it's an exercise in consensus building. Likewise, it is not reasonable to expect moral solutions to arrive at something "perfect and complete". Practically relevant morality will be fuzzy and everchanging, just like judicial systems.

I am quite sad that so many philosophers and members of the public seem reluctant to accept this challenge at overhauling the millenia old stagnated academic debates. If they don't participate, engineers will "solve" these problems themselves, perhaps choosing ease of implementation over moral considerations.