This isn't a trivial problem. These patients are often asymptomatic, and objective physical exam findings are not easy to elicit or entirely non-existent. At this point, the only thing a doctor can do is screen every patient coming through the door for cancer via CT, MR, mammography, colonoscopy, etc., but this doesn't translate into clear cut clinically meaningful benefits that helps patients live longer. You often end up irradiating non-cancer patients and creating new cancers, or you selectively diagnose non-aggressive cancers (length-time bias) and unnecessarily put patients through surgery, chemotherapy, and radiation (i.e. over-treatment). Some fraction of these patients then go on to live shorter lives as a result of the over-treatment. This particular issue we're discussing is more of a medical science problem than a healthcare system problem.
Anecdote from med school: on my surgery rotation, I had a ~70 year-old patient recently diagnosed with early stage colon cancer on a screening colonoscopy. He had undergone chemotherapy and was now about to have the cancerous part of his colon removed. Surgery was a success, and the patient recovered in the hospital for a few days before we sent him home. The day after he goes home, we found out he died from a massive pulmonary embolism, likely a post-operative complication from surgery.
Alternatively, had this patient not had a screening colonoscopy, we never would have found his early stage cancer. Based on his cancer stage, he likely would've lived another 10–15 years without treatment (age ~85-90). Instead, our interventions killed him a few months after diagnosis.
> The records of all births, deaths, and surgeries are known.
Actually, this is not known (at least in the US). There is no central medical database or significant communication between hospital systems. As you can imagine, this is problematic for obtaining meaningful data. In order to answer the questions you raise, researchers have to begin collecting new data prospectively or look retrospectively at the incomplete records contained within their hospital system. The retrospective method is easier and probably more common, but there are serious biases with a retrospective approach, including small sample sizes and non-random sampling (many US hospitals serve specific demographics).
I find this somewhat astonishing and deeply troubling. How can patients or doctors truly make informed decisions if they are not in fact well informed? This is not right at all.
The other party responsible for the lack of centralized data is the EMR vendors (Epic being one of the largest). They're incentivized to lock-in their clients (i.e. hospitals) by making any kind of data sharing at best a pain in the ass and at worst entirely impossible. For large health systems, I've heard the cost of changing EMRs is in the tens to hundreds of millions (USD).
Besides the detriment to research, this does lead to real patient harms. Where I went to med school, there were two major hospitals systems about 5 mins from each other in a major city. I worked in the emergency department at one hospital where I would frequently see patients coming in after car accidents and heart attacks. But often, some of these patients had received all of their care at the hospital 5 mins away. We had zero records for these patients. Meaning, we had to work them up from scratch and any important medical history would be unbeknownst to us. This obviously led to poor care and duplication of medical tests, which can be harmful (e.g. radiation from CTs). And requesting medical records from the other hospital was infeasible in emergencies. Realistically, these records were only available 9a–5p on weekdays via fax at the other hospital's leisure.