Study of Theranos Medical Tests Finds Irregular Results
nytimes.com
nytimes.com
"There was as much variability between LabCorp and Quest as there was between Theranos and LabCorp or Theranos and Quest."(1)
Also a link to the actual paper in case anybody feels like reading it from the horse's mouth (2)
(1) http://www.techinsider.io/theranos-study-compares-blood-test...
Yeah, so these are the kind of people I want to trust with my personal health?
The problem isn't the tech. The tech is close enough to accurate to be massively profitable. The problem is their reaction to criticism. It erodes any and all trust in the product.
When you are dealing with a company built upon measurement of the invisible, trust is the most valuable commodity. I need to trust that the numbers from your machine are reported as true.
I need to trust that you aren't lying and covering up data to make a buck...
How do you know? It says that apparently the methodology could be flawed due to things in the blood clustering together
>you aren't lying and covering up data
What? From reading, admittedly only, this article here, it sounds at least reasonable to complain given the hugely negative press when it is also true that "the study authors made no attempt to find out which measurements were right". So if I understand this correctly, the new method showed different results from the old method, but nothing was done to show which was right, and it was clearly insinuated the new one was wrong. I would complain, too. But perhaps you know more about this incident.
From the reported independent results.
+-5% compared to Quest and LabCorp at a fraction of the cost is good enough to make a profit.
>It sounds at least reasonable to complain
It makes sense to complain in Silicon Valley...
But it makes you look like a slimeball trying to cover your ass in the medical/scientific industry.
Industrial science is different than tech. In tech, if it works, it works, and nobody really cares how. In fact, it is in your best interest to hide how you did it, because that is your IP.
In industrial science, if it works, but you fight people trying to open the black box, it makes you look like you're faking data.
kmonda, your statement requires some very important clarifications
1) your phrasing implies that there was a flaw in the methodology of the journal article rather than specifically noting that Theranos asserted there to be flawed methodology (A claim which other, independent, experts interviewed by the NYT did not agree with)
2) The blood clumping comment is, similarly, not an accurate representation of the article. The Dr. who noted the potential blood clustering issue was not noting it as a problem with the study methodology, he was noting it as a problem with Theranos' technology. This would contribute to error in Theranos' measurements, and the shift seen in the study. It would not contribute to error in the study itself. Theranos' assertion of error was about the vein blood draw causing error in the Theranos readings.
Further, the same Dr. whos words you are implying damages the credibility of the study is, instead, quite blunt about his opinion of Theranos
>“This represents just another effort for a failing company to cover up bad press for a technology that is not ready for prime time,” Dr. Frederick L. Kiechle, medical director for clinical pathology at Memorial Healthcare System in Hollywood, Fla., said about Theranos’s letter to the journal.
>Dr. Kiechle said it was known that cells in blood “tend to clump together and travel in groups,” so the tiny amount collected from a finger prick might not be representative.
EDIT: in the interest of transparency, I edited one sentence of this for clarity.
That's too reductive.
The WSJ cannot "forget to mention" a manuscript that wasn't published when their original articles came out.
The manuscript in question, an observational study, shows that Theranos has higher variation than standard labs for some tests, and that those tests are those that someone familiar with the field might suspect that a finger prick might have higher variation (e.g., lymphocytes).
The power of this manuscript is limited (in part because of Theranos not actively participating in peer-reviewed science like it's peers), but at a high level, it would hint that a physician sending a patient out for some particular labs would be wise to apply extra scrutiny to those lab values if the testing was by Theranos.
If true, this raises two questions in my mind:
1. Were the non-Theranos machines approved for testing diluted blood samples? The technology may be approved only for blood samples with normal concentration.
2. Were the non-Theranos machines qualified to test blood samples collected by any method? Some researchers think that capillary blood collected with a fingerstick may be too contaminated to produce accurate results.
In other words, was Theranos negligent with respect to federal regulations for use of these machines? Someone with clinical lab experience may want to weigh in.
Were personal computers replacing mainframes nuts?
> It's far more painful to take blood from a finger than from the MCV in your cubital fossa (arm).
Why didn't you just call it an arm?
> Then you have to go around touching things all day with a band-aid on your fingertips,
I've never had a band-aid for a finger-prick and do them constantly for checking blood sugar. Doctors never did it either.
Personal computers never replaced mainframes. Personal computers replaced minicomputers and workstations.
The difference is meaningful, even in this context: Minicomputers and workstations were high-end systems dedicated to things like relatively small-scale file storage, graphics processing, network tasks (servers, routers), and other workloads which required neither high throughput nor long up-times. They mainly required more RAM, CPU, and, sometimes, specialized hardware than it was feasible to put on a home system of the era.
Mainframes, on the other hand, had become specialized as high-throughput high-availability systems, running large databases and supporting dozens to hundreds of users at a time, for years on end. They'd grown entire specialized architectures dedicated to handling that kind of workload, and were only getting moreso as time went on.
My point is, if Theranos is the PC, it's only going to replace some aspects of the market it's moving into, and, from this article, it might not even do that. It's just too difficult to play catch-up with the extremely specialized realms, even if you have technology which is an 80% solution for 90% of the problems.
If you could really replace 10's of mls blood draws from the arm with a few drops (< 1 ml) drawn from the fingertip with no loss of clinically usable information, it would actually be a major breakthrough. There are distressingly large number of patients in hospitals requiring once a day (or even more frequent) blood draws to maintain correct treatment. The volume of blood and availability of veins and pain threshold can become actual, serious problems very quickly. This is doubly true in children.
If such tests were somehow found, they'll be noble-prize worthy scientific breakthroughs. Somehow Theranos is not giving off that vibe sadly.
However a result that is wrong, but looks correct does not set off any alarm bells. You'd better hope that the physician requested a battery of tests and in correlating the results, sees the outlier. Otherwise you're potentially screwed.
In running tests on actual people, saving money comes far behind the overriding concerns around quality of results.
What prompted my comment was, "While most of the results were similar for the three labs, 12.2 percent of the measurements reported by Theranos were outside the expected normal range for healthy people, compared with 7.5 percent for Quest and 8.3 percent for LabCorp."
To me that sounds like at least some of the tests may be erring towards false positives. More research needed.
One classic example is a pregnancy test. You're going to find out eventually, and the impact of a false positive is huge, so the test errs toward false negatives instead of false positives.
This video does a good job explaining it (and is where I learned most of what I know): https://www.youtube.com/watch?v=UF1T7KzRnrs
Well yes, I wasn't suggesting jumping into treatment before verifying with the expensive, higher specificity test.
Even with pregnancy, the impact of a false positive isn't huge when the next step is just to take another test - the same test that would've been done anyway if the first one wasn't available.
Though the potential savings probably aren't great with pregnancy tests because they're already cheap, and I'd imagine the overall incidence for people taking the test is pretty high.
So on that first paragraph, are they implying that the journal should have provided Theranos with a copy of the manuscript for review and comment? Maybe I'm not reading that charitably, but that's not how the system works, and I don't think scolding the journal (page 3) over manuscript decisions is an appropriate response.
The second paragraph is strikes me as a bit rich considering the reputation Theranos has for secrecy. As best I can tell, they still haven't published the more detailed information on their technology that they promised when the WSJ first broke the story?
Were I the editor, I think my response might be along the lines of "you raise some excellent points, and so we invite you to submit a research manuscript for accelerated review".
I suppose they are popular due to some combination of schadenfreude and the morbid fascination that comes from watching a slow motion train wreck. But I find these articles empty of anything remotely interesting let alone any edifying content. They are just a monthly sign post for the accumulation of disparaging comments about Theranos.
On one hand, it's possible that they're just too big to fail, considering the massive amount of capital they've raised and all of the vested interests put into the company. On the other hand, Enron comes to mind, which did end up failing and was fairly huge as well.
[1]: http://www.wsj.com/articles/theranos-has-struggled-with-bloo...
Chances of that are slim to none, they are damaged goods at this point. A number of existing customers has already cut ties to avoid further association and any future would-be investors would be very wary of putting more money in without iron-clad proof the technology is solid.
My opinion is that Theranos should still be a small biotech company operating in R&D mode. They had no business going to market with a methodology and a product that were unproven and entirely hidden from scientific peer review. After the dust settles and they reorganize, that’s probably what will happen. Theranos should also allow their investors to pull out without any consequences.
I'm amazed that they (or the people who contracted them) are not getting sued. If they mess up a blood test, someone could get mistreated and die.
By calling out the lack of peer review, I'm referring to the science behind the use of fingerstick sampling. When you extract blood from a vein with a sterile needle, the blood is uncontaminated. When you stab someone’s finger with a blade or sharp pin, presumably you have lacerated hundreds or thousands of capillaries. The blood that emerges is contaminated by tissues inside the finger and by oils and bacteria on the surface of the finger.
It’s possible to do simple tests like iron level with this blood and get accurate results. More complicated tests are another matter, and require scientific investigation.
Is this so? I don't see how one could affect the other.
That makes it a lot harder to get LPs to pitch in. Still, even today there is a glut of money chasing far too little viable start-ups so as far as I can see if you have a solid proposition it should not be hard to find funding. If you are a bs artist then it may get a little harder, and that's a good thing (especially for those entrepreneurs that do have something to offer).
Although I have my doubts Theranos blood tests are as acurate, I feel the study was pretty flawed.
Could you elaborate on why? My expertise is limited to an EMT certification that expired half a decade ago, but isn't obvious to me why this would be the case.
Blood is constantly circulating, and even a veinous blood drawn doesn't remove an appreciable amount of blood compared to the total flow.
When a you stick a needle in a persons arm you often trigger a flight-or-flight repsponse. It's actually fairly similar if you prick their finger. The problem with this study is that one was done after or before the other. It depends what you look for, but for example white blood count (which was tested) would/should change subtly after a blood draw.
The amount can vary (or not vary) depending on the person. For example, when I get my blood drawn my arm swells, it's not actually that I am allergic, but my body reacts in said way.
Regardless, my only comment was that the study seemed to be on somewhat shaky footing.
[1] http://www.frhg.org/documents/Lab_Manuals/Blood-Collection-A...
Interesting. What mechanism or phenomenon are you proposing would be happening? Why do you think sampling order has more influence than the sample method (venipuncture v. finger prick)?
> Further, when the baseline (i.e. other labs) also have varying results, its fair to call into question the whole study.
Why? That's an expected outcome when dealing with multiple testing labs.
Furthermore, it's hard to imagine how results could be dramatically altered. How would having a previous finger prick blood test for say glucose level, cholesterol, triglycerides, blood count (CBC), calcium levels, blood urea levels, or the presence of antibodies possibly change the results of a subsequent test?
http://techcrunch.com/2016/03/31/government-report-details-t...
Basically: How do you justify your belief that their sample size is to small?
There are calculations to do this. 60 patients is, while small, pretty standard for a study like this, and decently powered. The authors also used the appropriate techniques to deal with small numbers in particular strata when they were controlling for the variables they mention.