The higher level question is how should we respond when machines, trained on facts and data, systematically make decisions that go against our sensitivities?
The challenge we are going to face in this AI era is that the training data can be biased or include some morally/politically incorrect information. Someone can intentionally manipulate the data and make us feel that the response from the machine is trustworthy and factual, which can be devastating if it is used by a dictating government. Mind the alternative facts!!!
You'll be hard pressed to find a single fact that would be considered so by the entire human population. There are people who are seriously disputing that the earth is round.
Well, it's sort of a very slightly lumpy oblate spheroid... it also ties into whether we agree on what "round" means.
There's a knowledge problem involved - when answering a question you really have to understand the knowledge and intent of the speaker, establish a conversational frame.
The first years of university are often spent learning certain "facts" and theories, which are technically incorrect (but useful), which will be torn down and shown as approximations later.
Even things that look like "hard" facts are very often a matter of perspective.
We can UNintentionally manipulate the data too even when we are trying our best to be honest.
Suppose we have a program that approves or denies loan applications. We input the acceptable default rate, and it makes the approval decisions. Then, we notice that the software disproportionately denies applications from a few demographics. What's the right reaction?
Or, what if the software makes hiring decisions, where the objective function is "hire people that will, in aggregate, maximize the corporate stock price."
Now suppose again we have a machine that rate load applications. In the past when you write your occupation as AI scientist, probably it won't make a difference than others. However, same occupation in the past few years, even you haven't got the default data to retrain the machine, a human would know the risk is significantly lesser than the majority. The problem? While we think the machine is at inferring future outcome, it is not in some important cases which we are largely unaware of.