When an AI finally kills someone, who will be responsible?

Yeah, when an accident happens you still need to figure out who has financial liability for the damages. And you don't want a "tragedy of the common" situation where no one takes responsibility for improving things. Just picking someone to be "it" gives them some incentive to go out of their way to prevent future accidents.

Edit: that said, excessive finger-pointing is bad too. If everyone thinks that the one person deemed responsible is the only one who has to fix things, they're not going to work on it themselves. The NTSB seems to have the best model for this.

When you remove accountability, incentives to keep the system performing as intended become distorted. This is basic game theory. It is dangerous to imagine a human engineered item can be infallible.

While modern society does have both a litigious issue and a cowboy mentality issue, I think the consistent focus on accountability-by-lawsuit is mainly denial.

I'm selling that short, it is more involved. But take the average person off the street and say "we are going to make something using science, and that thing will do things so you don't have to." Do they think about things like "what happens to the world if routine tasks are universally cheap?" About how this would impact their life in a fundamental way?

They sure do! And the brain connects this to primal fears of the unknown, the anti science culture we've had since the atomic bomb, and just an inability to fit this within their schema. The brain shuts down these dangerous thoughts and focuses on "what can go wrong? ". After all, if it is wrong, then there is no point in thinking about those uncomfortable issues.

So we bounce between "who do we sue?" and "could it turn on us?", both of which fit in nicely with our preexisting schema. Irresponsible scientist meets impractical pipe dream. Both are legit issues, but so are plenty of issues being ignored.

People aren't consciously doing this, but it's happening all the same. People aren't even asking these questions to find solutions. You can tell by seeing how often ot how deeply they've considered it. Usually the immediate response will be some disaster scenario (e.g. car plows into kids for no good reason) but they have no parameters, no follow up. The brain found the question so that it could check off the box saying "don't need to think about that".

This may sound like I'm being elitist, but this sort of thing is but into humanity. Overcoming it in one area doesn't help you with others. My theory is that we're only actually conscious a very little bit of time, and the conscious minds natural goal is to figure out how to turn back off. That we are evolutionarily moved to keep this high caloric activity to a minimum. Brain says "we have a thing the lizard brain doesn't have any existing reflexes for, turn on self awareness! Activate backfill rationalization! " And we are aware, do a projection of "what if"s, provide expectations to the lizard brain to start new 'mindless responses, and then the need for awareness is over and we slip back into zombie mode. That's just my theory, but it applies to all of us. Some more or less than others, but to everyone. A lot.

> But in cases where there is no relation, no intention, do we really need to find the responsible? Shouldn't be better to focus on how to avoid and how to correct this "bug" ?

These two questions together are kind of incoherent. We need to find the responsible party, because otherwise there is no incentive to fix the bug.