Surely the only rational reason to care about liability is you get paid in the bad outcome.

Let's assume you get paid 0$ for AI mistakes -- no liability at all. Then your EV is roughly:

(AI accuracy - human accuracy) * (cost of accident) - (1 - human accuracy) * (human payout)

Now I don't know about you, but even with pretty big numbers on the human payout scale and relatively small differences, I would not accept 10 million (presumably paid to my next of kin) to die. Does lead to weird situations where I may prefer humans for minor medical procedures where I can take the risk -- but only in edge cases.

Or is there some weird punishment thing going on here?

Why can't the AI be held liable?

If I pay a company to do something, and that company uses an AI to do the thing, and the AI messes it up, why isn't the company liable?

Not true. The company deploying or using the AI (not the frontier AI lab building it) needs to make sure that the AI works. Just like anything else. Not sure why this is complicated.

This isn't going to compute with the AI evangelists on here.