>This is bound to eventually lead to a disaster when someone deluded into trusting the "AI" does some stupidity it suggests without passing it through any reality check.

Or, the AI is plugged into a machine because efficiency (aka the shareholders need even more profits), and it does some stupidity. And they will be deployed carelessly, because that makes money. And people will suffer.