One thing that people often overlook when talking about health insurance is that, economically, "health insurance" is really the combination of two completely separate products, only one of which is actually insurance.
Insurance is about pooling risk. It smooths risk across states of the world (e.g. the universe in which your house burned down, and the universe in which it didn't but your neighbor's house did instead). It reduces the variance between those states, and in exchange, it charges a premium for making the outcome more predictable. It is not there to save you money. The expected value of all insurance is negative, and that's intentional.
By a true insurance model, routine care and elective surgery would be priced the same with insurance as it would without, because there is no risk involved. The reason it isn't it because we want to subsidize these, to enable people to obtain care they otherwise wouldn't be able to afford. We use "insurance" as a vehicle for this, but really, that's not insurance at all. It's a redistribution.
So it's perfectly understandable that a primary physician who is dealing with routine care and doesn't accept insurance would be able to charge patients less (and still walk away with more money) than one who uses insurance.
The reason you don't see this everywhere, though, is that policies are making it harder and harder for physicians to run private practices entirely (whether they accept insurance or not). The majority of physicians are now employed, largely by hospital networks which are required to accept insurance (in some cases legally, and in others by virtue of the fact that they need to cover their Medicare losses[0]).
So, the place you are most likely to see the direct care model are with physicians who are catered to the incredibly wealthy. That's not to say that a direct care model couldn't work for people who aren't, but the way our system is set up doesn't facilitate it.