This statement is misleading.
"Instinctual and observational impulses" is poorly defined, but the low latency cognitive systems I think you are describing (such as object categorization) are inputs to higher executive functioning systems which determine behavior.
The fact of and ability to "override" generalizes too much about how these low latency systems are prioritized a-priori. In other words, some of them seem to be able to be able to be re-prioritized fairly easily (positive/negative response to specific olfactory stimuli), while others approach impossible without motor system intervention (Not reacting to severe pain stimuli).
It's categorically misleading to separate these from all other functions of Perceive > Process > Respond as all decisions are all built on the same reinforcement learning foundations.
By programming them this way (avoiding programming in human frailties for the purpose of better "fitting in"), I think we can more quickly expose and correct certain areas where us bags of mostly salt water act suboptimally.
The bigger point I'm making is people think AI acts in some form of posthuman reason. At this time no such thing exists: decisions are still bound to human bias, and we should at least consider optimizing with regard to the rules of a modern society.
Humans are constrained by the knowledge that they would dread for the rest of their life the knowledge that they optionally and intentionally took an overt action which resulted in the death of an innocent person (to save five, but they would know they killed one when they could have done nothing and had "clean hands"). Machines don't need to suffer from this.
I would hope that if we program safety-controlling systems that we wouldn't replace the compare and conditional-jump with "halt and catch fire".
In the problem statement, an actor has control over whether this 1 person or those 5 people die. If they choose red, then 1. If they choose blue, then 5. (Failing to make a choice is choosing blue.) If after playing the scenario, they knew someone would use a Men In Black pen on them to completely forget what happened, is there a rational argument against choosing red?
(I realize this might read like trolling, but I genuinely don't find the original problem a difficult one if you eliminate the human guilt aspect.)
Actively choosing an action to deliberately kill someone else could be weighed negatively by the machine, it would depend on the programmer.
I don't think you can apply the word "rational" here, if you don't take a position of utilitarianism, not deliberately killing the person could be axiomatically rational. It's not just about guilt.
The brakes had suddenly failed approaching an intersection and the car had the choice of either running into a group of five pedestrians crossing the street or slamming into the concrete wall to your right, killing only you. Which is the correct answer?
Now suppose it was not you in the car but your wife and two daughters. Would you want your car, the car you bought and paid for, to sacrifice your family to save the lives of five strangers? What if it were only four strangers? Or ten?
There is no right answer. But as humans we're biased to favour, if not our own lives, the lives of our closest family members over strangers. I, personally, would never buy a car if I knew it would make the choice to sacrifice my family to save strangers.
Concretely, the car should take the course of action to minimize the global harm based on the knowledge it has. If that's killing me in the event of a brake failure, so be it. The spectacularly unlikely probability of that is outweighed by the million more times when it and others like it will make a less selfish decision than today's human drivers.
> I, personally, would never buy a car if I knew it would make the choice to sacrifice my family to save strangers.
The point I find interesting here is "this consumer preference, if common, confers a propagation advantage to systems which behave selfishly".