I'm the author. A question I have is: how did so many prices ever get negotiated in the first place? What kind of systems are in place to do this kind of micro-negotiation?
Many of the private health providers are for-profit and lobby against rule changes that would reduce complexity and save the system money. It know this may sound glib, but if you are trying to understand the US healthcare system and something seems strange, usually it's because it makes someone money and they'll fight hard to keep it that way.
This is almost certainly an anti-competitive move. By keeping many rules and regulations, you need more staff to deal with them - and smaller insurers have fewer patients to amortize those salaries over.
And it's not just some money but very BIG money they make.
So your claim of public + private being the issue makes no sense when almost every country has that.
For procedures covered by provincial health programs, doctors in private practice (GPs, at least some surgeons) give the service to the patient and then bill the government for that amount. For services that aren't covered by provincial health programs, GPs/surgeons bill the patient.
Provincial health care programs cover anything medically necessary. My general understanding is that you cannot pay to have a service done which is covered by a provincial health program. The intent is to prevent the establishment of a two-tier health care system.
It's probably less complicated than the US system, but it's still complicated.
But, when publishing, they omit the context and just dump every negotiated rate. Because it's technically compliant, but keeps things opaque.
They basically denormalized all the dimensions.
Imagine you have a function which takes 5 arguments and returns one value. You could give me the source code and let me run this function. Or you could give me a mapping of every possible combination of the 5 inputs to the returned value. The former could be quite small, but the latter would be a massive number of rows.
If I understand correctly, in this case, that function's source is highly distributed in wetware. It's about as closed-source as it gets; nobody has anywhere near the full source. Each hospital is its own fiefdom!
But even if you had like 10M rows of pricing and then gave a 2% discount to entity A, 3% discount to entity B, 4% discount to C, etc.
You could publish these discount rules.
Or you could just multiply the 10M rows by the number of different entities giving 10*n M rows.
And then let the consumer of the data try to figure out the rules from the output...?
When it comes to large parties, multi-practice groups, health systems, etc, an overall fee schedule or charge master for an existing institution is typically not renegotitated line by line every year but as incremental changes from the previous. Many/most of the parties involved have been working together for decades, some even longer.
Many plans administered by familiar names like Anthem are actually funded and controlled by the large employers the plan services. In those cases the employer plays a role in defining what will and will not be covered and what will be paid and the insurer is a middleman (acting as a third party administrator).
Hospitals are an entirely different system. They have much more negotiating power and if an insurer has a customer that goes to a hospital emergency room outside of their contract, the insurer has to pay outlandish rates. So it is in the insurer's interest to make a deal. They achieve this by inventing different 'products' with different amounts of 'coverage' for different premiums. Each of these 'products' had their own negotiation and their contracts.
Price transparency is the first good thing that has been mandated. However, this misses the mark. The focus is the patient, not the insurer, the hospital or the physician. Accordingly, patients should be allowed to submit their explanation of benefits and their bills-this is the data that reflects the true cost of healthcare. All of the numbers provided by hospitals, insurers and physicians has been massaged and buried in a forest of minutiae.
The end result is that you might end up with an individual doctor having to work with the insurance company for pricing, so the same procedure can cost vastly different amounts at hospitals down the road from each other providing the same level of care. To make it worse we also have laws preventing healthcare providers from providing prices upfront, out of a fear that people will forego necessary care they can't afford.
Edit: seems like this changed 01-01-2021, now we do have some price transparency laws - https://www.cms.gov/hospital-price-transparency
What are these laws? This seems so backwards - I know personally I have put off medical care in my past because I had high deductible insurance, and no guarantee that the bill I'd get wouldn't wipe me out, and no way to price shop. Paralysis of unknown.
https://www.cms.gov/hospital-price-transparency
Now assuming the hospital is compliant the information should be available. To be fair my understanding of the argument for the old law was that you didn't want a hospital with a big sign out front saying "Broken arm repair: $10k" and having people not go in for it when there might be some financial aid they could get afterwards
Which often struck me as kind of weird - the rooms with shortest wait time are probably most expens
The real economy has way many more prices than this one - from each store of anything in the country that negotiates from straws to bread. The difference is that these ones happen in a system that has a paper trail from the doctor, to the insurance, and this admin burden is only (apparently) worth it because the vast majority of money in healthcare goes through tax-advantaged insurance.
Cash based payment should suffice for 50~70% of healthcare expenditues and it would have more prices and not have expensive and abusive billing processes.
Also! What did they do before they could store 100TB of pricing data? How has pricing (and care quality) changed as a result of being able to do this type of thing?
Maybe you have a couple one off negotiations for high volume procedures, but even still the source data could be several orders of magnitude smaller than the dumps.