>Suppose these models are found to be reliable enough for certain mission critical work (healthcare, defense, etc.) and they lead a user down a path that results in harm or death? Who's at fault? Suppose it becomes a liability of the company offering the model? How does the insurance market respond?

So much this. I think a lot of people overlook this aspect so much. I wonder if we'll see a corporate push to make people think of individual instantiations of LLMs as being their own entities, for scapegoating purposes. It's completely ludicrous, but you know they'd just love to point a finger at Machine #73583 and make that take all the blame.