AI will always need humans in the loop to intervene. Remember the old IBM motto: 'A computer isn't accountable so a computer should never make a management decision'.
The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.
A poetic sentiment, but does not hold in reality. Anyone who has spent more than ten minutes being a square peg in a round hole can surely recall their frustration when a bureaucrat is refusing to do the common-sense thing, because The Computer is telling them something that is complete and utter nonsense.
Once The Computer is part of any human process, its decisions, be they inscrutable, insane, or simply stupid are not to be questioned. The line drone that is using The Computer has no agency to countermand it, the line drone's manager has no agency in rebuilding, or re-configuring it, and the executive that authorized the system will never even be made aware of the problem.
These problems exist when The Computer is simply executing the rules the programmer fed it, and will persist in the same way when The Computer is executing an inscrutable neural network. Except it'll be worse, because neural networks are capable of executing even more tasks poorly.
-----
In theory you are responsible for, and control the heating element, but in practice, we're the society from Idiocracy that cooks everything by sticking it into the microwave, and pressing the 'Popcorn' button on it.
I'm much of a mind that Frank Herbert's thought experiment had the right doctrinal approach to the question of artificial intelligence. Thou Shalt Not Make A Machine In The Likeness Of A Human Mind.
Bureaucracy-induced frustration predates computers by centuries.
A sufficiently large quantitative difference turns into a qualitative difference. We're becoming a people that, at work, are turning all of our thinking over to machines.
To a degree. (See: piercing the veil.) Also, corporations need an authorised signer, who under current law must be a natural person.
Having algorithms and AI models decide how to profit regardless of how it harms humans is the inevitable next step.
Besides, it's predicted on accountability being something a large company would want. There's already a whole industry around accountability-washing as it is: management consultancy.