$5 device tests for breast cancer in under 5 seconds: study
studyfinds.org
studyfinds.org
Actual publication: https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/Hig...
Experiment size is literally N=21, with 4 healthy participants, 3 in-situ breast cancers, and 14 invasive breast cancers.
N=21 might as well be useless in my opinion. You can't draw any meaningful conclusions about statistical power of this test; if your priors were 10% for breast cancer, after taking this test, your posterior probably remains unchanged.
The new Theranos is here again
Theranos was actual lies. TFA is just being honest about a low sample size.
Maybe be a great, honest work, brilliant step?. Yes.
But please don't claim that your model is reproducible if you didn't ever tried seriously to reproduce it first. What if the results are just a random effect?. What if the test says just positive most of the time? The test negative response has been tested in a control group of 4 people? This is 20 bucks well spent.
--At this moment-- isn't different than other thousand projects that seemed too good to be true, took the money and soon vanished. I hope to be wrong.
Maybe they found something worth testing and just need more funding to actually do the tests?
I have no problem with early free speech about "I found something interesting" as long as you're honest about the sample size and don't misrepresent it.
These folks are just stating that, "so far, this looks promising".
At least, thats my read. YMMV.
It's very unfortunate, because it is technically possible, just very difficult to achieve, even in academia (so it won't be done first in industry).
This was the absolute worst part of Theranos, is that they deter others from trying to make headway in the space at all.
The issue is concentrations and the presence of them in sufficient amounts to be detected reliably and consistently.
edit: typo.. words are hard.
There are hand held sequencers for your phone now (minIon) that are very cheap and work. There are also large multimillion dollar bench top machines on the same technology from nanopore. Both are very new technology compared to what we used in early 2000s, and offer various different features and certainties from the analysis.
However, it is foolish to say that the minIon is impossible. It works. It's just less resolute than other systems.
This is true for many systems which people cry "Theranos" about. Their issue was lying. That was the problem.
Low concentrations will NOT be reliably detected. That's part of why such a comparatively large sample is taken when you go for the traditional testing.
I'm not even going into the simple fact that once chems are introduced into the sample to generate reactions, the sample can't continue to be used.
Exactly how many times do you think we can split a single drop of blood?
Edit: good talk, even if we disagree. :)
"Exploratory study shows promise" is better. When n=21, flipping a coin as about as accurate.
Few understand this.
I suspect it is deliberate nonsense.
But is that your intuition talking?
If you claim 90% accuracy and I want to be 95% confident with the standard alpha of 0.05 you need 38 trials.
(a) We should do more tests to increase N
(b) If a $5 device tests you positive, maybe you should go get checked out for $5000 or whatever the doctors charge you (because insurance often only pays AFTER "shit happens" and often does not pay to test "whether shit might happen"), that you wouldn't have thought of doing otherwise.
If this sample data is randomly sampled it looks like it will be a fairly high precision test for invasive cancers, with room for false negatives. You would have had to have gotten very unlucky to see such a difference in distributions even on a small sample like this. But sure, let’s get more samples.
One thing I'd note, though, is that the paper's title is "High sensitivity saliva-based biosensor in detection of breast cancer biomarkers: HER2 and CA15-3". My understanding is that sensitivity was never really the problem with breast cancer detection - it's specificity that is the real challenge with all types of broadly-deployed medical screening tools.
This is why it’s better to have tiers of confidence for actual policy decisions and not just confusion matrix results frankly.
Like if you run this test and you’re in a cohort that is 99.999% likely to be cancer, that’s useful info. It’s not a problem, per se, if the test instead comes back with an answer of mixed certainty. We just need to be comfortable with getting neither a positive nor a negative prediction; nor expecting it to cover all cases.
The policy driving thresholds on tests need to consider the weights of each possible outcome to determine what to do.
A normal human fallacy is to focus on the thing you understand, that is easy to understand (e.g., easy to quantify), and overlook the difficult issues that are far more important. There is much more, of much more importance going on in most of these studies, and in their statistics and validity, than the sample size.
I first took stats in high school and we read a bunch of valid small sample size studies so I don’t know where people are failing to get educated.
There were so many factors going into whether a study was good.
Famous example: Lady tasting Tea [1]. N=1 or N=8 depending on how you look at it. Still significant.
This is the one that proves that it's possible to tell whether you added the milk first or second.
> Now there are 8 billion people, so it's worth 16 times less.
That is not how statistics works, at all.
> In fact, this is why I always trust science in small towns more. In Colma, for instance, you sample 500 and you have covered 33% of the people.
These are the sentences that made me think this had to be satire (presumably you want "science" to apply to places outside of Colma...), but in all honesty these days it's really hard to tell.
To all who criticize and ridicule someone who would like to have more samples, why do you think 21 is such a perfect number in this case? Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much?
This is the point that you are fundamentally misunderstanding. Nobody is making the argument that "21 is such a perfect number in this case". The rest of your sentences ("Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much?") seem to point to a belief that people are pulling numbers based on a finger in the wind.
The whole point of (a lot of) statistical testing is that it allows you to come up with a specific number that determines how likely a particular result is due to chance. That is what the p < .05 "standard" is about - it's a determination that the results have less than a 5% likelihood to be due to random chance (though that 5% number used as "significant" is just basically pulled out of thin air, and p-hacking is another topic...) That is, I and the comment I replied to aren't making the argument that 21 "is such a perfect number". We're making the argument that even with a small sample size it's possible to determine the relative error bars, with precision, using statistical methods, not just pulling a number based on feels. Yes, larger sample sizes reduce the size of those error bars. But often not in ways that are "intuitive". You have to do the math.
None of what I wrote above is meant to imply that statistical tests can't be misused, or that they often require assumptions about the underlying population distribution that may not be correct.
Effect size is going to be important when talking about clinical significance, not just statistic significance, too.
I also want to point out that we have no stats on sensitivity, specificity, or diagnostic odds ratio, which are all clinically relevant to physicians deciding when to test and how to interpret test results.
The good news is that it's a non-invasive, low cost test which plays a factor in clinical decision-making.
It's only 1 in a million, but at about 10 the same I'd start to figure the coin wasn't fair.
You can find my published papers here:
https://scholar.google.com/citations?user=5DdrMc8AAAAJ&hl=en
You forgot the end of the story: After all that, it's done and published, and then a random person with no expertise and who barely read the paper posts on HN: the sample size is too small - as if you were in your first week of statistics 101 - and therefore the whole thing must be invalid! :)
I dunno... that was me then, I guess: hard core academic with a drive/compulsion to publish good work.
I admit to being amused by comments here about statistics by people who wouldn't know a Bonferroni correction from a bonfire.
There are two other issues with this paper [1] that I quickly see. Their main figure lacks errors bars. It’s pretty clear to me that the groups would overlap quite a bit. More numbers would make this problem clearer (either to narrow error bars or clearly show an overlap). The lack of error bars across the paper make me think they didn’t do any technical replicates, which is also a problem.
I’m also not sure a one way test is correct here, but I’m also not entirely sure how they are measuring the data. In this one way analysis, all you can tell is if one group is higher than the other. When the data are so unbalanced, what you don’t see is what the predictive value of the test is — false positive/negative. That’s the real issue here. It looks like you’d have a really high false negative rate, as the cancer samples have a much wider range than the controls. This is the worst thing you can have in a test like this.
Finally — this paper was published in the “journal of vacuum sciences and technology B.” There is no way this paper got a valid peer review to make these claims. I don’t know anything about this journal, but I doubt it has much experience with cancer testing.
[1] https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/Hig...
The error bars are not really needed. Your eyeballs are doing just fine. Yes, they overlap. Yes, it would likely have a high false negative rate.
A false negative rate is not a problem for a cheap test. This test is not meant to replace higher quality and more invasive tests. A high precision, low recall test still has significant value. You merely have to accept that a negative result tells you very little.
If we imagined for instance, that the false positive rate is very low despite the overwhelmingly larger population of people without cancer, then this would be of enormous value.
But, that's not really the point. They aren't trying to diagnose cancer vs. healthy.
Error bars here are absolutely necessary. Two reasons: First, you want to know the approximate ranges for each group in Figures 3 and 5. Not showing them is misleading. Secondly -- you actually also want error bars for each patient sample. I'd expect for there to be at least three replicates for each saliva sample to show that the strips are able to consistently measure a known value from each sample.
I also mis-read part of the paper the first time. For the HER2 cases, there aren't 4 negative samples -- there are 20. There is only one positive sample. Part of the problem is really how they are presenting the data -- it is not all clear what they are testing. But, there is only one HER2+ sample in the mix.
One... N=1.
Samples include:
* Non-cancer: 4
* In situ cancer: 3
* Invasive cancer, HER2-: 13
* Invasive cancer, HER2+: 1
What you'd really like to show is that the HER2+ patients could be differentiated from HER2- patients. Which, does look really good, but with only one HER2+ sample, you really can't tell much. (And the presence of so much signal in the HER2- samples raises some very interesting biological/mechanistic questions).
Note: I'm not trying to say that the authors of the study are wrong or are trying to deliberately mislead people. There is so much here that could have been corrected to make this a much stronger paper. To me, this seems like a paper where the authors are likely engineers and not that well versed in biomedical statistics. The paper is published in a physics journal, so the journal itself is not a good place to make some of these arguments.
Is the idea of a non-invasive test worthwhile? Yes! Absolutely. But they didn't show that it was a good test of clinical utility. They showed that it could measure differences in protein concentrations from saliva. That's not nothing, but that's it. Now, if that is an appropriate way to differentiate patients is a completely different question and requires substantially more testing (and orders of magnitude more patients).
Complete Nonsense
> That is not enough to be able to adequately know range of measurable values in the population of patients w/o cancer.
True. But it is a promising initial signal that the distribution of non cancerous folks is probably very different from the invasive cancer folks. The effect size here is huge. “Range” is less interesting than “Distribution”
> Error bars here are absolutely necessary. Two reasons: First, you want to know the approximate ranges for each group in Figures 3 and 5. Not showing them is misleading.
You don’t really need error bars when you’re showing all of a small number of data points. But sure whatever
> Secondly -- you actually also want error bars for each patient sample. I'd expect for there to be at least three replicates for each saliva sample to show that the strips are able to consistently measure a known value from each sample.
Would be nice. Sounds like they did ten measurements per.
> What you'd really like to show is that the HER2+ patients could be differentiated from HER2- patients. Which, does look really good, but with only one HER2+ sample, you really can't tell much. (And the presence of so much signal in the HER2- samples raises some very interesting biological/mechanistic questions).
I’m not clear why you’re saying HER+ vs HER- is the important difference here.
What I said ("it doesn't tell you anything about the ability to differentiate between the three populations") is quite correct. This study shows that there is a difference between the groups of samples tested with their HER2 test strip, with a one-way p-value of ~0.002.
I'm not convinced that the samples are representative of their populations. The number of non-cancer samples too low.
>* You don’t really need error bars when you’re showing all of a small number of data points.*
Error bars are visually helpful ways to show that the group values overlap. Which, in this case, they do (I did replot this data to confirm).
>* Would be nice. Sounds like they did ten measurements per.*
This is for one test. They sampled the test strip 10 times. I mean they should have tested each sample on at least 3 different test strips to get a mean value for the sample. This is a paper that is trying to say that their test strips are accurate, so it would make sense to test them multiple times.
>I’m not clear why you’re saying HER+ vs HER- is the important difference here.
I'm not sure what they are trying to claim in there paper... are they trying to say that they can diagnose breast cancer (which would requires many more biomarkers), or are they trying to say that they can differentiate between HER2+ and HER2- cancers (which would be more appropriate for a HER2 test).
The other biomarker has even more overlap, so not sure how helpful that would be.
Really, I think they are also missing an opportunity -- the bigger use for me would be in longitudinal testing. If they could show changes in signal over time for a particular patient that corresponded to treatment status -- that would be a great use for a cheap non-invasive test.
You claimed this was the implication of an unbalanced study. I’m sorry but that is a complete non sequitur. It is nonsense. Most of everything else I disagree with but isn’t nonsense.
> I'm not sure what they are trying to claim in there paper... are they trying to say that they can diagnose breast cancer (which would requires many more biomarkers), or are they trying to say that they can differentiate between HER2+ and HER2- cancers (which would be more appropriate for a HER2 test).
They are simply showing different distributions of tests for different populations and observing that, foremost, the invasive cancer ones have a significantly different distribution; and that it’s shows promise as a test for the future.
In this case, and my lack of understanding in both bio and stats is showing here, we're trying to develop a test for breast cancer. An ideal test will have high sensitivity and specificity, and a tight confidence interval for both of those numbers. This way we can be confident that a positive/negative test actually moves our priors meaningfully in either direction.
I guess my question/comment is more on the fact that I don't see how any of the results shown actually translate, as the headline suggests, into a cancer test of high power. The priors for any kind of cancer is pretty low from what I can find, so we need high power tests in both the positive and negative direction to meaningfully effect health outcomes. I can't find any CI numbers in the paper, which may just be me not reading it closely enough, but it doesn't help my confidence.
If the test is very cheap, it’s probably a great test presuming you get to see the dice roll itself.
The cost and invasiveness of the test is important.
The larger sample sizes are a ... sort-of "proxy" ... to overwhelm underlying latent correlations.
The whole thing is actually a subtle sort-of generalization of the "Prosecutor's Fallacy".
So skepticism with the small sample sizes is absolutely warranted, unless some strong evidence is shown indicating mechanism-based independence.
In terms of applicability, it depends on whether or not ELISA is in fact the current standard of care, but it could be useful in low-resource settings where you don't have lab personnel trained to carry out those assays, and drug choice is also restricted by limited availability.
Additionally, there's a point-of-care argument as well. Since breast cancer does benefit from early detection, I can see a future in which biomarker testing is a more regular thing, and high saliva concentrations are flagged. At the very least as something worth bringing up at one's next appointment or wtv.
HER2 testing is done on all breast cancers as it affects treatment choices though the majority of breast cancers are not HER2 positive. (HER2 is also expressed in some normal tissue (notably cardiac) and is also seen in other cancers as well).
> ... In Fig. 5, the test results for detecting CA15-3 of the human samples are displayed. The digital reading decreases from the healthy group to the invasive breast cancer group, indicating an increase in CA15-3 concentration. The median, the range by disease status, and overall p-values analyzed with the Kruskal–Wallis test for the CA15-3 test are listed in Table I. The overall p-value for CA15-3 is 0.005, indicating that this device provides an efficient way to detect the salivary biomarkers related to breast cancer.
My goal in pointing out the difference was not to be snarky. It was to point out the very real statistical consequences. Any model can be accurate on a sufficiently biased dataset, but what matters once a screening test hits the real world are the precision (positive predictive value) and negative predictive value. These are the hurdles that the test will have to pass to see widespread adoption.
I realize this is an early technology and I think it should continue to be explored, but I would anticipate that if compared head to head with screening mammograms, it would be inferior.
For patients with relapsed disease, this kind of technology would be neat to non-invasively re-assess biomarker status but as a screening tool, I find it lacking (and certainly a positive screening will require dedicated imaging and biopsy anyway).
This is WRONG, it is used as an indicator, NOT a diagnosis. In fact, I have personal experience that CA15-3 is not always in indicator of anything. You first measure a baseline, and use it for reference.
Also there's no HER2 expression in TNBC.
Can this be used for possible early detection? Maybe, but it will not be an exact science.
Discusses sensitivity but not accuracy or rates of false positive/negative.
and then in 2022 for detecting oral cancer: https://pubs.aip.org/avs/jvb/article/41/1/013201/2866658/Hig...
Dare I ask why?
>Instead of using the transistors as the sensors, which need to be disposed of after each use, a system with a reusable printed circuit board (PCB) containing a MOSFET and disposable test strips were employed. In this approach, synchronized double-pulses were applied at the gate and drain terminals of the transistor to ensure that the channel charge does not accumulate, and there is no need to reset the drain and gate paths to mitigate the charge accumulation at the gate and drain of the sensing transistor for sequential testing. With the double-pulse approach, it only takes a few seconds to show the result of the test, due to the rapid response of the functionalized test strips and resulting electrical signal output. As an example, the LoD has been demonstrated to reach 10−15 g/ml and the sensitivity to 78/dec for COVID-19 detection. Similar approaches have been used to detect cerebrospinal fluid (CSF), cardiac troponin I, and Zika virus.27–30
> In this work, use of this double-pulse measurement approach to detect HER2 and CA15-3 in saliva samples collected from healthy volunteers and breast cancer patients was investigated. The voltage output responses of the transistor correlated to the HER2 and CA15-3 concentrations, detection limits, and sensing sensitivity were determined.
I mean… even if we were talking some special hardware, that’s still PLC territory.
My reading was this isn’t $5, but could be made to be $5.
This is clearly a prototype.
It tests for biomarkers in the saliva. Possibly not outright crazy charlatan territory.
Could certainly use a larger sample size though, especially given that one of its bragging points is "fast and cheap!"
I'm guessing it is expensive to jump through all of the hoops. Without a patent, why would a company pay for this?
If there is no patent, and no in-place infrastructure already producing the device, then yes, it's unlikely to see rapid scale up by manufacturers.
Though if it's really that cheap to produce, I'm sure it will come onto market in some form (whether through charitable foundations or otherwise). All assuming that the device is actually as efficacious as the study implies (with a small sample size).
There are a little over 6,100 hospitals in the US at any given time. That means roughly $30,000 in cost to produce to provide every hospital with one, using the cheapest of cheapest parts. Presumably, you'd want higher end components and things like a shell casing to protect from spills, static, etc. maybe the cost is $120k.
You could sell these for $100 each, and maybe make $5 million. That's a decent amount of money for a tiny business, but a pittance for even a small business that needs to pay salaries, lawyers, etc.
In reality, they'd probably go for much more- let's say $10k each, plus $100 for each test strip. Still easily in range for the budget of most US hospitals.
The real question is, if breast cancer is suspected, is this test any better than imaging using equipment the hospital already has?
Can it detect all types, or the degree that it is progressing?
I suspect the utility in a clinical setting is not high enough to really change clinical practices any.
> “[...] cost-effective, with the test strip costing just a few cents and the reusable circuit board priced at $5,” Wan says.
That is cool: the 5$ isn't even the cost of the test, it's the one-time cost of your lab equipment. Of course, per sibling comments, the efficacy has yet to be seen, but even a few percent more early detections due to frequent testing would be a win
This is not the current medical thought on early screenings for various cancers. It used to be and I was confused about it until very recently. Indeed the medical community is still wrestling with the issue of screening harms. The consensus is shifting that screening should only be done if there is an existing condition or symptom or family history.
https://www.cancer.gov/news-events/cancer-currents-blog/2022...
Most cancer treatments are really nasty. Thus false positives are really bad: you destroy someone's quality of life. The earlier cancer is discovered the better chance that we can use a less harmful treatment (if only because of smaller dose of the harmful drugs)
The current breast cancer screening is an xray - which itself causes cancer (about 1 in 3000 cases of breast cancer discovered by xray wouldn't have got breast cancer in the first place without the screening - the screening is still wroth doing if you are at risk, but don't do it if you are not at risk).
Breast cancer can be deadly, but if caught early it is easy to treat (normally).
The medical concern generally isn't should we test all women for breast cancer, but when do we start testing and how often should we test. If this test is safer than an xray and sensitive enough it can be useful. Avoiding current breast cancer tests is good.
I had no idea the risks associated with xrays for breast cancer screenings were that high. Do you have a source (for that 1:3000 assertion) I can read?
False positives on a screening test are bad because you follow a screening test up with a confirmatory test (a biopsy for cancer) - sometimes the procedure for the biopsy results in additional complications and even death in rare cases (and if it's a false positive, a patient goes through all of that for a benign finding).
I want to be very clear that oncologists are not going to start cancer treatment on the results of a screening test, you need confirmation.
I was imagining this cheap self test like covid self tests: you don't start taking chemo, but it's an indication to see a doctor tomorrow if not today and get it checked out properly
But you do raise a good point about overtreatment which I had forgotten while writing my previous comment. Whether that applies to diseases that are reliable to confirm manually and will simply kill you if you're late to the party is another matter, but I had forgotten to take it into account at all
Through hole DIP array? This is a prototype and it makes me skeptical of the whole thing. There is nothing a single $5 FPGA couldn’t do. So why not start there? I suspect because the people that made this doesn’t know electronics or programming well - but then also didn’t find someone that did.
This was put together the way long way, and it’s strange to me.
The experimental design seems very small scale though. 17 cancer positive samples (of which only 1 was HER2 positive), 4 control. Since the strips are focused on HER2 detection I read this as "in 1 out of 1 samples, our test detected HER2 overexpression" but maybe I misread it.
[0] https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/Hig...
But this is what I do know: Given that the University of Florida is a public institution and the governor of the state is anti-science and promotes anti-science policies and practices, my immediate reaction to this paper’s news is negative. Reasonable? Probably not. I can’t be the only one who would react in this way.
This matters for two reasons. First, each of us has limited time and capacity to care about things, to investigate them more. Second, if we don’t recognize our biases, we may ignore important and useful information.
1) These are not clinical biomarkers used for detection of cancer now, and in fact, they are known NOT to be clinical biomarkers useful for detecting cancer. 2) This is a publication focusing on the device/method, not the clinical application. It is published in a journal of vacuum science, not a cancer biology or medical journal. 3) There are several inaccurate things in the paper, one being that they state that current technology requires 1-2 weeks to measure either biomarker. Wrong, clinical tests exist today that can perform those immunoassays in minutes on large automated analyzers. 4) This isn't fraud, it's just a really typically overhyped report of a novel device/meausrement strategy (and it's not that novel) that targets a biomarker that has a role in cancer, and then some mass media picks it up and says that they have "the test for cancer". This happens all the time. 5) This should maybe be considered a proof of concept about the electronics of their detection strategy, since the immunoassay component is known (immunoassays to both biomarkers are not only published, but commercialized) and the clinical use of the biomarkers is not at all diagnostic for cancer or useful for screening.
An MRI machine cost arounds $350.000 on average. https://www.blockimaging.com
Also, diagnosis would typically be done using a mammography. The cost of such a scan is lower - around $100[0].
[^1]: https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/Hig...
[^2]: https://www.thermofisher.com/elisa/product/ErbB2-HER2-Human-..., note that I didn't shop around for necessarily the best prices.
The thirty through hole parts made me the think the same thing.
This device may be total crap, I don’t know, but “trust the system” isn’t a great way to navigate the American medical system. Other countries have it better, or at least different.
There is a lot that could be improved about the US medical system, but Nobody has to bribe doctors and insurance to sell tongue depressors.
This is the opposite. If the state of the art was a fancy tongue manipulation machine that cost $30k, used licensed consumables billed at $100/patient, and did a bunch of non-essential things that doctors found convenient once in a while, would someone selling a box of sticks get anywhere?
flag this and ban the website
I mean yeah, I get it, it's a prototype and a finished product will be on a $2 ASIC to drive the correct signals and etc. But I'm not up to speed on affinity sensors vs. traditional ELISA tech so <internet shrug>.
> Synchronous voltage pulses are sent to both the electrode of the strip connecting to the gate and drain electrodes of the MOSFET. The drain pulse is applied for around 1.1 ms at a constant voltage. The gate pulse starts at 40 μs after the drain pulse and ends at 40 μs before the end of the drain pulse.
> the antigen-antibody complexes undergo stretching and contracting, akin to double springs, in response to a pulsed gate electric field. This motion across the antibody-antigen structure, corresponding to the pulse voltage applied on the test strip, induces an alteration in the protein's conformation, resulting in a time-dependent electric field applied to the MOSFET gate. Consequently, a springlike pattern emerges in the drain voltage waveform due to the external connection between the sensor strip and the MOSFET's gate electrode.
So they shake ‘em just so, and listen to the response…
ICs are perhaps variable timing & pulse-shaping logic?