I think there are two reasons for these "inexplicable" pricing differences.
One is that different insurance companies specialize in different things. For example, if you want to offer the best rates to good drivers, you might actually want high-risk drivers to go somewhere else, or at least compartmentalize them so that your prime clients can still pay favorable rates. But if you do that, another company will show up and specialize in the customers you're walking away from. There's enough competition that no single firm can have it all, and there's definitely a lot of divide-and-conquer going on.
I also suspect that there's a multitude of ways to end up with a specific risk profile, so there's plenty of more or less random divergence on the tail end. Some insurers get where they want to be by giving discounts if you install a monitoring app. Others if you're in a "safe" profession. Etc.
> The math of insurance is based on having the biggest, most diverse risk pool to sop up any poor bets.
Sort of? Sometimes? There are diminishing returns, especially for car insurance where claim limits are fairly modest - I doubt it makes much of a difference if you have ten million versus a hundred million customers. A state-run monopoly has a comparatively weak incentive to make the math work, because it can always count on being propped up by the taxpayer - and if not, it can charge you more or offer worse service, and you can't do much about that.
Private monopolies share some of the same problems, so I wouldn't root for either. Thankfully, the car insurance market in the US is pretty competitive.