Galleri: Interpreting results from GRAIL's multi-cancer early detection study | Hacker News Reader

Galleri: Interpreting results from GRAIL's multi-cancer early detection study

arsham.substack.com

69 pointsaxg1110 comments

It’s worth noting that a test’s performance (i.e. sensitivity/specificity) is tunable. That is, Grail is making a decision of where to set its threshold for a positive result, in effect selecting where it wants to be on the ROC curve. For this version of this test they’ve chosen to keep the specificity very high, with the tradeoff being a low sensitivity. However, as this article alludes to, a low sensitivity can be okay if there isn’t any other screening modality — the alternative is essentially 0% sensitivity. (Like for, say, pancreatic cancer.) They can change the threshold to have a higher sensitivity but then will drop the specificity, and in settings of low prevalence (i.e. cancer in asymptomatic population) you quickly end up with a crazy false positive rate.

There are already ongoing trials with newer versions of the test (and many competitors not too far behind), but as you can imagine the challenge of demonstrating early detection gets harder the “earlier” you’re talking. (Longer study times, larger populations.)

In any case, highly recommend people read the linked study[1], particularly the confusion matrix which provides a quick overview of the test’s ability to identify the cancer’s source from just the blood. (That’s cool.)

Also, the Galleri test is already on the market in the US. (Via prescription by your MD.)

[1] https://www.sciencedirect.com/science/article/pii/S092375342...

> (and many competitors not too far behind)

From what I have heart (from someone at a cfDNA start-up), competitors have the solid problem of competing with a monopoly here. Grail is Illumina and Illumina will have a quasi monopoly on diagnostic sequencing until their core patents run out (hopefully around 2025?) and BGI and others will join the field. For a diagnostic test, you would want to integrate a number of preparation steps with the sequencer but cannot when the company whose help you'd need for that is competing with you.

Separating Grail out to be bought again for a much higher price was a strange move from Illumina, or at least a bad financial decision in hindsight.

Still, if the 2025 year that you write is true, other companies may be already able to use the patent in a non-commercial way (just for clinical trials and future potential revenue split)

Separating out Grail allowed the company to take on outside investment, much of which I imagine flowed back to Illumina in the form of sequencing instrument and consumable sales.

The IP that Illumina is using the block the sale of MGI instruments expires in 2024, I put together some notes on this here:

https://41j.com/blog/2021/05/the-next-few-years-in-dna-seque...

There will probably be a drop in sequencing costs at that time, as there are a few players set to build on expiring IP. Singular Genomics (recent IPO) is one of these, they could possibly be on the market earlier than this (by avoiding some of the last IP to expire).

There's no non-commercial exception for patents, so I don't believe patented approaches could be used in clinical trials. But on the scale of clinical trials, Illumina sequencing would not be prohibitively expensive...

I agree it was an odd decision at the time. Grail is very much built on Illumina’s platform and I don’t see them building anything otherwise. If Galleri is successful, Grail will become Illumina’s biggest customer and the two could become completely interdependent.

In the medium term, I think there’s an opportunity for a company to build ctDNA screening on a different sequencing platform. That could present unique advantages against Grail which is very much tied to using Illumina.

> Grail is very much built on Illumina’s platform

I’m not sure this is true. GRAIL’s technology is (a) a protocol for efficient bisulfite conversion of cfDNA to infer cytosine methylation, and (b) proprietary algorithms for using that methylation information to infer presence of cancer. Both (a) and (b) are agnostic to the actual sequencing machine (e.g. Illumina): the end product of (a), namely cfDNA fragments with non-methyl C’s bisulfite converted to uracils, could be sequenced on any machine[*] capable of reading short DNA fragments, not just Illumina’s machines. (b) just requires methylation information, which again, is sequencer agnostic.

[*] companies making such machines will likely become abundant once Illumina’s patents expire in a few years

Grail will have to end-to-end validate any changes to Galleri, especially any change of sequencer. Given that any Illumina competitors based on the same IP wouldn't hit the market until 2024 (extremely optimistic) I'd be surprised if Grail were able to launch a non-illumina Galleri before 2026. A company that is locked into the Illumina platform for the next 5 years is very much dependent on Illumina in my eyes.

> you would want to integrate a number of preparation steps with the sequencer

I think this is unlikely, Grail like everyone else probably just do standard sequencing runs with their own prep.

The issue is that Illumina make something like 10x on sequencing consumables.

So if Illumina were to sell a sequencing based test, they have large margins they can cut into to undercut anyone else.

That said, Illumina's dominance in sequencing is probably ending in ~2024 anyway (when basic patents expire).

Grail is not Illumina, it's a spinout that they want to acquire back in. Most likely this is because Illumina see that:

a) This Grail experiment is working out. b) They're going to lose their dominance in sequencing, and they want to acquire downstream applications.

I would be curious to see the positive predictive value (PPV) of this test (i.e. given that you have a positive result, what is the probability of having the disease?), which is a very different quantity from sensitivity, which measures the probability of having a positive result, given that you have the disease. The answer to this question will depend on whom the test is administered to.

For example, given a test with 95% sensitivity and specificity for a disease that affects 1% of the population, if we administer the test uniformly to the population (i.e. without accounting for any prior probability of having the disease), the probability of actually having the disease given a positive test result is quite low, since the number of healthy people who get a positive test will be much greater than the number of sick people who get a positive test, owing to the fact that the vast majority of people are healthy.

The math for this is just Bayes’s rule:

p(disease|pos) = p(disease and pos)/p(pos) = p(disease and pos)/(p(disease and pos) + p(healthy and pos))

The highlighted term in the denominator is what lowers our PPV. We expect it to be much larger than p(disease and pos), since the fraction of healthy people in the population is much larger than the fraction of diseased population.

Expanding the joint probabilities, we have:

p(disease and pos)/(p(disease and pos) + p(healthy and pos)) = p(pos|disease)p(disease)/(p(pos|disease)p(disease) + p(pos|healthy)p(healthy))

Since our sensitivity and specificity are both 0.95, we have

p(pos|disease)=0.95 p(pos|healthy)=0.05

and since we administer the test uniformly, the prior probability of having the disease is the overall incidence level in the population, p(disease) = 0.01, so p(healthy) = 0.99

Plugging these numbers in, we get

0.95*0.01/(0.95*0.01 + 0.05*0.99) = 0.16

In other words, you only have a 16% chance of actually having the disease if you get a positive test.

The only way to improve on this number is to (a) construct a test with much higher specificity (hard!), or (b) only administer the test to people who are already more likely to have the disease than the baseline probability (i.e. pick a better prior, much easier!).

Given the figures in this article (99.5% specificity, aggregate 50% sensitivity) and an overall annual incidence of new cancer cases of ~450 per 100000 [0], we get a PPV of ~30% for GRAIL. This is probably fine for certain cancers, which can be followed up with minimally invasive additional screens, but too low for others, which would require invasive biopsies. I assume the actual PPV is much higher, since this test presumably will only be administered to a subset of people with a higher probability of having cancer, whose incidence rate is much greater than 450/100000. As I said before, I’d love to see what that number actually is.

[0] https://www.cancer.gov/about-cancer/understanding/statistics

An interim analysis of Grail’s Pathfinder study [1] suggests a PPV of 43% [2]. This may be a mixed population of their high and normal risk cohorts - it’s not clear to me from the abstract. They call this a conservative estimate, which seems quite fair, given that the “discordant” positive cases are really only potential false positives at this point. These patients may actually harbor tumors that become detectable further into the study or on follow up.

Even with a mixed risk study population, a lower bound estimate of PPV in the 40s seems like a big win for Grail.

1 https://clinicaltrials.gov/ct2/show/NCT04241796

2 https://ascopubs.org/doi/abs/10.1200/JCO.2021.39.15_suppl.30...

Reply on news.ycombinator.com