Theranos Has Struggled with Blood Tests
wsj.com
wsj.com
http://www.newyorker.com/magazine/2014/12/15/blood-simpler
I mean "nice" as in, a good read...it's a flattering profile of Holmes -- she comes off well, as a genuinely passionate person, and even if she had stayed in school, it sounds like she would've still made a great impact.
But the article also raises concerns that seem to be corroborated by the OP:
> Clarke argues that finger-stick blood tests aren’t reliable for clinical diagnostic tests; because the blood isn’t drawn from a vein, the sample can be contaminated by lanced capillaries or damaged tissue. Holmes strongly disagrees: “We have data that show you can get a perfect correlation between a finger stick and a venipuncture for every test that we run.” When I asked for evidence, I was sent a document by Daniel P. Edlin, Theranos’s senior product manager, titled “Select Data.” It purported to show favorable results from numerous comparison tests. I asked Edlin if the tests had been conducted by an independent third party. He replied by e-mail: “The clinical tests were conducted by a combination of Theranos and external labs,” but he wouldn’t say which ones.
Um, OK. I don't know much about this testing process...but...what trade secrets are being protected by hiding the methodology and source of the comparative test results?
I don't doubt that she is sincere in wanting to build this business and take over the blood testing industry. But Theranos is looking more and more like a cautionary tale about why VC's and angels shouldn't give any Stanford dropout with little more than an idea and a dream a multi-billion dollar fictional valuation and tens or hundreds of millions of dollars in real cash to burn in the streets.
Wow. What a terrible response to get from company leadership. It should be completely appropriate for an employee to bring a concern to the attention of leadership based on a plain-English reading of a statute or regulation. Obviously, I don't know what kind of email the person sent, but if it was something like, "A plain English interpretation suggests that we are required to ... " then the right response is to thank them for bringing it to attention, and then engage people for an appropriate review. Every time I've dealt with a legal situation in a corporate environment, the plain English reading was ballpark a correct way to understand it, with occasionally some detailed nuances for which it was important to have legal input, but which did not radically change the situation. I've never run into a case where the required action was opposite than it seemed, or anything like that, although I'm sure surprising things come up from time to time.
I understand why they might ask people to stop discussing it, on a big company email list or something like that, but bringing it to the attention of the appropriate company leader, is exactly the right thing to do. Thank them and look into it.
The other parts of Theranos's media strategy, according to the WSJ, doesn't seem particularly friendly:
- Threatening to sue a widow whose Theranos-employed husband expressed doubts to her before killing himself.
- Going around to people who had been interviewed by the WSJ and asking them to sign statements (before the story was even published) that the WSJ had misquoted them.
- Holmes ignoring WSJ for _five months_. Then standing them up last week. The CEO is famous for being an inspirational, thoughtful speaker but can't even make time in _five months_ to give a bland statement of defense or explanation to the Wall Street Journal, on a story that strongly hints at violations of federal health regulations? It's one thing to refuse comment when a news organization ambushes you the night of the deadline -- it's inexcusable to hide for five months..._especially_ if the company isn't actually doing anything wrong.
Counsel also gives training about when to ask for legal advice and how to handle privilege and confidential communication, which doesn't sound like were followed here (or maybe never provided). As you say, it could be a liability during litigation. However, the response to that could have been "I've added our legal team for input on the issue you've raised. Please don't discuss legal issues except in privileged communication with our legal team." I guess that's easier to say and do when you have inhouse counsel. -g-
I would also hope that a thread like that would be deemed to be protected by privilege if counsel were immediately added like that as part of a request for legal advice. Perhaps someone more familiar with the boundaries of privilege could comment. From what I understand though, opposing counsel are usually pretty hesitant to attempt to pierce privilege, especially if there's a tinge of legitimacy on it. E.g., random guy raises legal concern to superior, who immediately involves counsel. Opposing counsel has to actually attempt to make the case that it's not privileged communication, which they are reluctant to do in such cases (but doesn't mean it can't happen).
http://www.degruyter.com/view/j/cclm.2015.53.issue-7/cclm-20...
1st Theme says: Theranos doesn't use Edison (thier in house testing device) & instead use regular equipment from companies like Siemens. This is a marketing problem because the company is saying its using one device but is actually using another one.
2nd theme: lab tests from Theranos differ from generally accepted standards. How do they differ if in fact they are using the same tests as everybody else? Is it just the general variability of lab results and similar variability could be find in quest diagnostics as well?
3rd Theme: almost all people say Theranos is dramatically cheaper than competitors. How is that possible when they are using the same equipment as everybody else for most of the tests? Is it a process innovation in operations rather than from Edison/better equipment tech? Or are they just subsidizing these costs and being cheaper and possible have bad unit economics?
The are two plausible storylines that can seem to reconcile these three themes is:
Storyline 1 (Negative)- Theranos seem to be doing a process innovation rather than an underlying equipment innovation. That process innovation perhaps includes diluting blood samples 1) to meet thier marketing promise of taking less blood 2) somehow taking less blood and diluting the samples to meet the standard for traditional equipments AND still lead to cheaper operational costs that lead to lower prices. But - somehow these diluted blood samples show more than normal variability.
Storyline 2 (not so negative) Traditional lab companies are ridiculously inefficient from an operations perspective. Theranos is able to take the same equipment as everybody else but because of thier operational efficiency make the end service dramatically cheaper. The variability in tests results is kind of standard in the lab testing market.
Am I thinking about this the right way or missing any big parts?
Theranos:
(1) takes a much smaller amount of blood, claiming they will use the Edison machine to analyze it
(2)(a) sometimes uses the Edison machine, whose accuracy is disputed
(2)(b) sometimes dilutes the sample to test using a regular machine. The dilution moves normal concentrations outside the range of detection of the machine, so when a 10x dilution has its concentration multiplied by 10, there are huge errors, often causing misdiagnosis
When Theranos receives a proficiency test, they don't receive a small amount of blood. They receive the amount of blood a normal lab receives. These are the only samples that they can run in the manner the standard machines were designed for.
Theranos competes on costing less and taking less blood. The doubts are all around whether you can get accurate results with smaller amounts of blood from a different body part.
From the article:
> The two types of equipment gave different results when testing for vitamin D, two thyroid hormones and prostate cancer. The gap suggested to some employees that the Edison results were off, according to the internal emails and people familiar with the findings.
> Former employees say Mr. Balwani ordered lab personnel to stop using Edison machines on any of the proficiency-testing samples and report only the results from instruments bought from other companies.
> The former employees say they did what they were told but were concerned that the instructions violated federal rules, which state that a lab must handle “proficiency testing samples…in the same manner as it tests patient specimens” and by “using the laboratory’s routine methods.”
Because Edison and the dilution procedure produce low quality results. Thus for a large number of the tests Theranos is less accurate than traditional methods.
> 3rd Theme: almost all people say Theranos is dramatically cheaper than competitors. How is that possible when they are using the same equipment as everybody else for most of the tests? Is it a process innovation in operations rather than from Edison/better equipment tech? Or are they just subsidizing these costs and being cheaper and possible have bad unit economics?
The above is possible because they are not currently focused on profit, rather market share. All that money that they have raised has had to go somewhere.
http://www.degruyter.com/view/j/cclm.2015.53.issue-7/cclm-20...
This is the same kind of bullshit I see in academia, where a new technology is promised to replace an old technology that works fine, but now they have a 9 billion dollar valuation!
I am not sure it is bullshit, maybe it's just competition?
Over time, companies within an industry become complacent, cut costs, and decrease quality of service. Somebody (a young idealist perhaps?) comes along and says, hey, we can do better! And because she believes in free markets, it must be that old technology (and indeed newer technology may have marginal benefit) is the culprit, not human greed or laziness or other deadly sin. So she goes, touts, implements and there is some success. However, then the giants will wake up to the threat. They will magically find the way to be more effective, as they always could have been, and they may even grudgingly implement some parts of new technology. And so the margin for the disruptor quickly disappears, and things will return back to the normal state of affairs, where companies can lazily collect fat paychecks for doing the necessary minimum. It's actually win-win for everybody, customers get a small technology/process upgrade, and she is now in the club!
The only minor problem is that this economic model is far from being optimal, but no one really worries about that except academic economists, since no one really knows how to attain the optimal model in the human society anyway.
Worst case is that their technology has a few percentage points better outcome, perhaps by selectively sampling candidates during clinical trials or combining it with an existing method, and everybody gets charged a ton more money.
I spent a number of years with one major company in this field, and back in the 1990s they already had a decade of R&D, and countless millions of dollars, invested in this area. It was a difficult area of research back then and it continues to be.
The idea that one scientist could come up with a concept to outsmart those companies is certainly possible and makes a very compelling investor story. But the reality is probably a little bit different.
Though in the UK the fractured nature of the NHS means I can't get my tests done at a hospital near work - I have to take a days leave.
I wonder if There a market for mobile blood testers who come to your work place to take the sample.
That being said, this particular lab company provides horrible customer service and our nurses probably spend more than an hour on the phone with them every week trying to troubleshoot very simple situations. For this reason, I've been hopeful about Theranos' potential to bring some meaningful competition to the market. We'll see.
With all these competing risks, this frequent testing strategy needs to be evaluated in randomised double blind clinical trials. Before that though, Theranos needs to submit their tests for independent validation in many thousands of patients, and publish the results.
I note this single study which compared Theranos' CRP to a conventional method [1]. The Pearson correlation was 0.85, which I don't think is really good enough.
This is an excellent point. Nevertheless, it's very hard to convince patients, and the public, that more tests do not lead to better outcomes. Think on every time there are revisions to the recommended screening criteria for breast or prostate cancer---it's good evidence-based medicine, but there's an outcry from the public every time.
In addition to your considerations of more testing being invasive and expensive, I'd also add that overtesting can lead to interventions that present a non-minimal risk to the patient and ultimately increase M&M.
This is exactly the problem! Nobody has sufficient information to judge Theranos's technology outside the company, and they're asking the scientific community and public to accept their claims at face value because they've submitted some documents to the FDA.
I can't believe the reporter didn't take Theranos up on its free offer to collect his blood and analyze it for personal health information that might be useful to Theranos...er, I mean, to the reporter.
Note the use of the word "current". Implicitly, they're admitting that the process has changed since the time of the former employees, and they're not disputing the process described by those former employees. Which means, that at some point they indulged in rigging proficiency testing.
It seems like disclosing that a sample is a test is more likely to negatively affect the accuracy of the test than improve it. The need for double blind trials is normally well understood. This is not an experiment, but as a test it seems like it would benefit from at least single blind: lab is unaware - accuracy tests are mixed randomly into the genuine population of testing requests. Kind of like how the TSA is occasionally tested by inspectors who bring all sorts of weapons through.
Typical PR drivel. It's also a bit rich to complain about scientific inaccuracy, and boast of opening things up to the FDA, when the Theranos testing platform remains to be thoroughly investigated in the peer-reviewed literature.
Then why doesn't the statement say what these facts are?
This reads more like an attack on the former employees who spoke up than a scientific rebuttal of the article. Which leads me to think even more that the article is right.
Seems like there would a lot more complaints considering the number of tests conducted, and that the relationship would have been soured/severed if the results were not living up to Walgreen's satisfaction.
You can resell a test so long as you follow the rules of the contract of the manufacturer and applicable CLIA and other FDA regulations. These are subject to audit etc... You should not report clinical results from tests not run in a CLIA or FDA approved way. So if a test is run on Siemens for reporting clinical results then the audit goes on that pathway. Diluting the pinprick blood to run on a test that requires larger volumes probably won't work either unless they've shown it does and that of course depends on the CLIA and/or FDA regulations around the test to prove equivalence. It seems like it won't work since the tests have specific input requirements (e.g. 100mL plasma vs 1mL). You might be able to go the other way.. but you miss the cellular and interstitial components that may add noise (e.g. potassium which cells bioaccumulate etc)..
Which other rules apply probably depends on interpretation of whether these are lab developed tests or FDA regulated and this area may require clarification etc... which will depend test to test etc...
Otherwise it sounds like they may have calibration issues which are normal. Outliers happen all the time. If the technology doesn't have some fundamental limit of detection based on the sample volume then they just need to keep iterating. Or the components of blood from the pin prick differ significantly from blood taken from the vein. The data may just be noisy. What that noise level is matters in terms of the detection limits. It is probably too early to tell. Internally they will be tracking this and trying to figure it out.
So this article seems like click bait, but what do I know.
https://twitter.com/MadMoneyOnCNBC/status/654674975798919168
Please provide an example where a bank "convinced" people to buy something it "knew" wouldn't perform. That would be fraud. I'm sure it occurs, but certainly not "quite often".
And, no, Goldman Sachs going short securities it sold as a market-maker isn't an example of such.
Really? What if your family member died because of inaccuracy of Theranos' results? Would you still cheer for them?
This whole article is extremely damning. Some stuff just makes your head spin:
>Carmen Washington, a nurse who worked at a clinic owned by Walgreens in Phoenix, says she began to question Theranos’s accuracy after seeing abnormal results in potassium and thyroid tests.
>She says she raised her concerns with the drugstore operator and Theranos’s lab director, asking for data to show that the company’s finger-prick testing procedures produced results as accurate as blood drawn from a vein.
>“They were never able to produce them,” she says. Ms. King says the company did show detailed testing-accuracy data to the nurse.
Anyway, if you ever need any tests done, stay away from Theranos.
https://www.google.com/search?q=Theranos+Has+Struggled+With+...
edit:
I had known about Theranos mostly for being a highly valued startup by a young college dropout involving something in the medical field...I assumed it was more related to the scientific research side...but in the article it says it's been offering tests to the public since 2013 (and that that appears to be the main potential source of revenue)...has anyone's doctor suggested the tests to them? It looks like they have a couple testing centers in Palo Alto but my general physician has never mentioned the option to me.
http://padlet.com/dpetersen/khmmi11wy8hz http://padlet.com/dpetersen/r6dg1vbxmz92
I don't know how an individual could evaluate the accuracy of these results, but the same goes for a non-Theranos blood test as well.
As far as I know, Theranos is in two places: Palo Alto and Arizona. It seems like Arizona is passing laws to allow testing without a doctor's order. California, you still need one.
If you are anywhere near Palo Alto and want to get some tests done, just message me, we'll sort something out :)
First link shows Vitamin D is very low and should be affecting your friends energy levels at the very least. Second link shows a 1500%(!) increase in AST in under a year and its way over limit.
One way to evaluate the accuracy is the sniff test, e.g. is the lab result congruent with how the person feels? Information on the internet around understanding your medical tests is pretty crappy unfortunately, but you can usually get to understand whether symptoms match. And if you have a doctor that gives you the time of day and helps you understand it, even more so. Your friend's AST result raises a question of whether there is an accuracy issue and should be investigated.
From WebMD [0] --
> High levels of AST may be caused by:
> * Liver damage from conditions such as hepatitis or cirrhosis.
> * A heart attack or heart failure.
> * Many medicines, such as statins, antibiotics, chemotherapy, aspirin, narcotics, and barbiturates.
> * High doses of vitamin A.
> * Kidney or lung damage.
> * Mononucleosis.
> * Some types of cancer.
> Many different conditions can raise AST blood levels, so other testing is usually needed to interpret an abnormal AST result.
[0] http://www.webmd.com/digestive-disorders/aspartate-aminotran...
His doctor thinks the high test may have been a result of hard work he was doing with a personal trainer, after a few years of not working out.
He had AST/ALT tested again a few weeks later, also with Theranos, and it was down dramatically (though still above reference range).
It does sounds like he could have been be a victim of their accuracy problems.
“They were never able to produce them,” she says.
Ms. King says the company did show detailed
testing-accuracy data to the nurse.
In the part that you excerpt, the WSJ reports the nurse saying that the company failed to produce data that would prove the accuracy of the test. The company (Ms. King) says that it did produce the data.
...How would you word that and the surrounding paragraphs?
The excerpt I quoted was isolated like a paragraph, but I don't remember a paragraph being 2 poorly written sentences.
How would I have worded it? For one I would have presented one side of the entire story, and then an opposing view-point (i.e. the view point of Theranos execs), instead of "person a said x. person y said x'".
I guess I'm sorry I'd prefer writing hidden behind a pay wall to be a little more elegant than what an 8 year old could write.
If someone expects me to shell out a subscription for their articles, the absolute least they should do is make sure the writing is above a 5th grade level.