Let me ask you this - can a company be held accountable? I.e. are you ok with the legal manner in which when I hire a company to provide me a service and they fail to provide it, or cause harm in the process, I can sue them, potentially in a way that would lead to their bankruptcy?

If so, I can imagine a potential future in which we have limited liability companies each run by a single AI (potentially on a particular physical computer). In that future, if you hired an AI to do a project for you, and it ended up deleting the production database, you'd be able to sue it, and get a payout and/or bankrupt it, which I imagine would then lead to an "antifragile" ecosystem whereby AIs adapt to be more careful.

Mine is usually "Measure Twice, Cut once (perfectly), aw hell, they're handed, I've got two left sides".