Why are cancer guidelines stuck in PDFs?

I’d rather have the pdf than a custom tool. Especially considering the tool will be unique to the practice or emr. And likely expensive to maintain.

PDFs suck in many ways but are durable and portable. If I work with two oncologists, I use the same pdf.

The author means well but his solution will likely be worse because only he will understand it. And there’s a million edge cases.

The OP will be pleased to know that they’re not the first person to think of this idea. Searching for “computable clinical guidelines” will unearth a wealth of academic literature on the subject. A reasonable starting point would be this paper [1]. Indeed people have been trying since the 70s, most notably with the famous MYCIN expert system. [2]

As people have alluded to and the history of MYCIN shows, there’s a lot more subtlety to the problem than appears on the surface, with a whole bunch of technical, psychological, sociological and economic factors interacting. This is why cancer guidelines are stuck in PDFs.

Still, none of that should inhibit exploration. After all, just because previous generations couldn’t solve a problem doesn’t mean that it can’t be solved.

[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC10582221/

[2] https://www.forbes.com/sites/gilpress/2020/04/27/12-ai-miles...

The fundamental idea here is that doctors find it difficult to ensure that their recommendations are actually up-to-date with the latest clinical research.

Further, that by virtue of being at the centre of action in research, doctors in prestige medical centres have an advantage that could be available to all doctors. It's a pretty important point, sometimes referred to as the dissemination of knowledge problem.

Currently, this is best approached by publishing systematic reviews according to the Cochrane Criteria [0]. Such reviews are quite labour-intensive and done all too rarely, but are very valuable when done.

One aspect of such reviews, when done, is how often they discard published studies for reasons such as bias, incomplete datasets, and so forth.

The approach described by Geiger in the link is commendable for its intentions but the outcome will be faced with the same problem that manual systematic reviews face.

I wonder if the author considered included rules-based approaches (e.g. Cochrane guidelines) in addition to machine learning approaches?

[0] https://training.cochrane.org/handbook

"At their core, guidelines are decision trees"

That's wishful and perhaps not even helpful as a goal. Guidelines rarely have the data to cover all possible legs of decisions. They report on well-supported findings, offer expert opinions on some interpolated cases, and perhaps list factors to consider for some of the remainder. If you reduced this to a decision tree, you'd find many branches are not covered, and most experts could identify factors that should lead to a more complex tree.

The reason is that branches are rarely definitive. It's more like quantum probabilities: you have to hold them all at once, and only when treatment works or doesn't does the disease (here cancer) declare itself as such.

Until the true information architecture of guidelines is captured, they will be conveyed as authoritative and educational statements of the standard of care.

In almost all cases, it's more important to reduce latency and increase transparency (i.e, publish faster but with references) than to simplify or operationalize in order to improve uptake. Most doctors in dynamic fields don't need the simplification; they rely on life-long self-discipline and diligence to overcome difficulty in the material, and use guidelines at most as a framework for communication and completion, i.e., for knowing when they're addressed known concerns.

Structured guidelines mainly enable outsiders to observe and control in ways that are likely to be unproductive.

Why isn't all human knowledge in one big JSON file? Guidelines are decision trees, but they're not written to be applied by rote, because the identical patients with identical cancers posited in the hypothetical don't exist. The guidelines are not written for maverick clinician movie protagonists to navigate the decision tree in real time while racing against an oncological clock, they're for teams of clinicians who are very well trained and used to working with each other, and who have the skills to notice where the guidelines might be wrong or need expansion or modification. That is, they're abstractions of the state-of-the-art.

Now it'd be nice if these could be treated like source code on some sort of medical VCS, to be easily modded or even forked by sub-specialists. But wetware dependencies are way more complicated than silicon ones, and will remain harder to discretize for some time to come.

It's not that the author's aspirations misguided, they're great. But I believe progress in this area is most easily realized as part of a relevant team, because what looks conceptually simple from outside the system only seems that way because of a lack of resolution.

Same reason why datasheets are still PDFs. It's a reliable, long lasting and portable format. And while it's kind of ridiculous that we are basically emulating paper, no other format fills that niche.

It's the niche HTML should be able to fill, since that was its original purpose, but isn't, since all focus over the last 20 or so years has been on everything else, but making HTML a better format for information exchange.

Trivial things like bundling up a complex HTML document into a single file don't have standard solutions. Cookies stop working when you are dealing with file:// URLs and a lot of other really basic stuff just doesn't work or doesn't exist. Instead you get offshot formats like ePUB that are mostly HTML, but not actually supported by most browser.

> With properly structured data, machines should be able to interpret the guidelines. Charting systems could automatically suggesting diagnostic tests for a patient. Alarm bells and "Are you sure?" modals could pop up when a course of treatment diverges from the guidelines. And when a doctor needs to review the guidelines, there should be a much faster and more natural way than finding PDFs

I have implemented this computerized process twice at two different startups over the past decade.

I would not want the NCCN to do it.

The NCCN guidelines are not stuck in PDFs, they are stuck in the heads of doctors.

Once the NCCN guidelines get put into computerized rules, they start to be guided by those computerized rules, a second influence that takes them away from the fundamental science.

So while I totally agree that there should be systemtticization of the rules, it should be entirely secondary and subservient to the best frontier knowledge about cancer, which changes extremely frequently. Annually after every ASCO (major pan-cancer conference) and every disease specific conference (e.g. the San Antonio breast cancer conference), and occasionally during the year when landmark clinical trials are published the doctors need to update their knowledge from the latest trials and their continuing medical education, which is entire body of knowledge that is complementary to the edges of what the NCCN publishes.

Having spanned both computer science and medicine for my entire career, I trust doctors to be able to update their rules far faster than the programmers and databases.

Please do not get the NCCN guidelines stuck in spaghetti code that a few programmers understand, rather than open in PDFs with lots of links that anybody can go and chase after.

Edit: though give me a week digesting this article and I may change my mind. Maybe the NCCN should be standardizing clinical variables enough such that the rules can trivially be turned into rules. That would require that the hypotheses that a clinical trial fits into those rules however, and that's why I need a week of digestion to see if it may even be possible...

Decision trees work for making decisions...

But they don't work as well as other decisionmaking techniques... Random forests, linear models, neural nets, etc. are all decision making techniques at their core.

And decision trees perform poorly for complex systems where lots of data exists - ie. human health.

So why are we using a known-inferior technique simply because it's easier to write down in a PDF file, reason about in a meeting, or explain to someone?

Shouldn't we be using the most advanced mathematical models possible with the highest 'cure' probability, even if they're so complex no human can understand them?

As a cancer researcher myself, I'd point out that some branches of the decision trees in the NCCN guidelines are based on studies in which multiple options were not statistically significantly different, but all were better than the placebo. In those cases, the clinician is free to use other factors to decide which arm to take. A classic example of this is surgery vs radiation for prostate cancer. Both are roughly equally effective, but very different experiences.

It's so much worse than you could possibly imagine. I worked for a healthcare startup working on patient enrollment for clinical oncology trials. The challenges are amazing. Quite frankly it wouldn't matter if the data were in plaintext. The diagnostic codes vary between providers, the semantic understanding of the diagnostic information has different meanings between providers, electronic health records are a mess, things are written entirely in natural language rather than some kind of data structure. Anyone who's worked in healthcare software can tell you way more horror stories.

I do hope that LLMs can help straighten some of it out but anyone whos done healthcare software, the problems are not technical, they are quite human.

That being said one bright spot is we've (my colleagues, not me) made a huge step forward using category theory and Prolog to discover the provably optimal 3+3 clinical oncology dose escalation trial protocol[1]. David gave a great presentation on it at the Scryer Prolog meetup[2] in Vienna.

It's kind of amazing how in the dark ages we are with medicine. Even though this is the first EXECUTABLE/PROGRAMMABLE SPEC for a 3+3 cancer trial, he is still fighting to convince his medical colleagues and hospital administrators that this is the optimal trial because -- surprise -- they don't speak software (or statistics).

[1]: https://arxiv.org/abs/2402.08334

[2]: https://www.digitalaustria.gv.at/eng/insights/Digital-Austri...

Cool tool. From my experience the PDF was easy to traverse.

The hardest part for me was understanding that treatment options could differ (i.e. between the _top_ hospitals treating the cancer). And there were a few critical options to consider. NCCN paths were traditional, but there is in between decisions to make or alternative paths. ChatGPT was really helpful in that period. "2nd" opinions are important... but again you ask the top 2 hospitals and they differ in opinion, any other hospital is typically in one of those camps.

I find the web(HTML/CSS) the most open format for sharing. PDFs are hard to be consumed on smaller devices and much harder to be read by machines. I am working on a feature at Jaunt.com to convert PDFs to HTML. It shows up as reader mode icon. Please try it out and see if it is good enough. I personally think we need to do much better job. https://jaunt.com

I know it's not the same, but in many areas we have this "follow the arrows" system in many guidelines. For some examples, see the EULAR guidelines with it's fluxograms for treatments and also AO Surgery Reference with a graphical approach to select treatments based on fracture pattern, avaliable materials and skill set.

I think that's a logical and necessary step to join medical reasoning and computer helpers, we need easier access to new information and more importantly to present clinical relevant facts from the literature in a way that helps actual patient care decision making.

I'm just not too sure we can have generic approaches to all specialties, but it’s nice seeing efforts in this area.

Gee, before talking about complex stuff like decision trees, how about we start with something really simple like not requiring a login to download the stupid PDF from NCCN?

The real problem is that the guidelines are written for humans in the first place. Workarounds like this shouldn't be needed, to go from a machine friendly layout to a human friendly one is usually quite easy.

And from what he says a decision tree isn't really the right model in the first place. What about no tree, just a heap of records in a SQL database. You do a query on the known parameters, if the response comes back with only one item in the treatment column you follow it. If it comes back with multiple items you look at what would be needed to distinguish them and do the test(s).

Software that gives treatment instructions may be a medical device requiring FDA approval. You may be breaking the law if you give it to a medical professional without such approval.

Why can this not just be a website? Isn‘t this a perfect use case for HTML and hyperlinks?

Forgive me if I'm mistaken, but isn't this exactly what the FHIR standard is meant to address? Not only does it enable global inter-health communication using a standardized resource, but it's already adopted in several national health services, including (but not broadly), America. Is this not simply a reimplementation, but without the broad iterations of HL7?

I parsed some mind maps that were constructed with a tool and exported as pdfs (original sources were lost a long time ago) and I used python with tesseract for the text and opencv and it worked alright. I am curious why the author went with LLMs, but I guess with the mentioned amount of data it wasn't hard to recheck everything later.

In the UK guidelines are published by the National Institute for Clinical Excellence. Guidance is available to all in html and pdf formats.[0]

[0] https://www.nice.org.uk/guidance

Profit.

I use these guidelines all the time in the PDF format for free, and I'd love to have these in a structured format. For $3000/year you could get 50 users access to PDF prescription templates to speed up their work. That's not bad, but it's still all PDF.

For the nice low price of "contact us for pricing," though, you could have EMR integration. They couldn't justify $$$$ for EMR integration if all this information is easily accessible.

https://www.nccn.org/compendia-templates/nccn-templates-main...

Because writers don't think about readers. PDF is one of the worst formats for science/technical info, but yet. I've dumped a lot of papers from arxiv because it formatted as 2-column non zoomable PDF.

The idea of adding hallucination to medical advice seems very dangerous.

> The whole set of guidelines for a type of cancer breaks down into a few disjointed directed graphs

Nothing undermines medicine quite so thoroughly as yet another astronaut trying to force it into a data structure.

The real question is: why is everything stuck in PDFs, and the more important meta-question is: why don't PDFs support meta-data (they do, somewhat). So much of what we do is essentially machine-to-machine, but trapped in a format designed entirely for human-to-human (also lump in a bit of machine-to-human).

Adobe has had literally a third of a century to recognize this need and address it. I don't think they're paying attention :-/

Would love to take a look at the code, in particular at how the data extraction and transformation is implemented.

As a side note, the German associations of oncology publish their guidelines here (HTML and SVG graphs): https://www.onkopedia.com/de/onkopedia/guidelines

The author makes a great case for machine-interpretable standards but there is an enormous amount of work out there devoted to this, it’s been a topic of interest for decades. There’s so much in the field that a real problem is figuring out what solutions match the requirements of the various stakeholders, more than identifying the opportunities.

You might be interested in checking out the WHO SMART Guidelines. Nothing on cancer yet AFAIK, but it's evolving.

Shouldn't the end goal be just to train an ai on all the pdfs and give the doctors an interface to plug in all the details and get a treatment plan generated by that ai?

Working on the data structure feels like an intermediate solution on the way to that ai which is not really necessary. Or am I missing something?

Excellent read. This consolidated and catalyzed my my spurious thoughts around personal information management. The input is generally markdown/pdf but over time highly useless for a single person. Thete would be value if it is passed through such a system over time.

The FHIR clinical reasoning specs provide machine-oriented formats to talk about diagnostic flows. But the actual science is going to be PDFs for a while, and translating them into machine-friendly formats is going to be extra work for years and years.

GraphViz has some useful graph schema languages that could be reused for something like this. There's DOT, a delightful DSL, and some kind of JSON format as well. You can then generate a bunch of different output formats and it will lay out the nodes for you.

Why is anything stuck in PDFs?

PDFs are just good for just one thing: printing. Data stored as PDF is meant to be printed and not being processed by any other means.

I have to ask: did the author contact any medical professional when writing this article? Is this really something that needs to be fixed, and will his solution actually fix it?

It seems to me that ignoring the guideline is a physician decision, and when it is ignored (for good or for bad), it is not because the guidelines are not available in json.

This is all predicated on the guidelines actually reflecting best practices

PDFs are a universal, machine readable format.

super cool post until i saw you used an llm to do this

Funny i just had the thought the other day about how we as a society need to move past the pdf format or even just update it to be editable in traditional document software. The fact that Google docs will export as a pdf and not have it saved in the documents is proof its gotten to a point of inefficiency and that's just one example

a decision tree is just a csv trapped in amber. share the actual data

WAIT ... Hole up... what have we here: https://www.nccn.org/compendia-templates/compendia/nccn-comp...

TLDR: The NCCN surely has a clean pretty database of these algorithms. They output these junky pdfs for free. Want cleaner "templates" data? Pay the toll please.

What we have here is a walled garden. Want the treatment algorithm? Here muck through this huge disaster of 999 page pdfs. Oh you want the underlying data? Well, well, it's going to cost you.

What we have here is not so much different than the paywalls of an academic journal. Some company running a core service to an altruistic industry and skimming a price. OP is just writing an algorithm to unskim it. And nobody can really use it without making the thing bulletproof lest a physician mistreat a cancer.

To my sentiment this is yet another unethical topic in healthcare. These clunky algorithms, if a physician uses them, slows the process and introduces a potential source of error, ultimately harming patients. Harming patients for increased revenue. The physicians writing and maintaining the guidelines look the other way given they get a paycheck off it, plus the prestige of it all, similar to some scenarios in medicine itself.

The natural thing to do is crack open the database and let algorithms utilize it. This whole thing of dumping data in an obstruse and machine-challenging format, then a rube goldberg machine to reverse the transformation, it's not right.

Anyway I mention this because there seems to be a thought of "these pdfs are messy lets clean them" without looking at what's really going on here.

> With properly structured data, machines should be able to interpret the guidelines.

Yeah, right. And then say "Die". /s

The guidelines shall be structured properly. It is not rocket science.