Google’s DeepMind made ‘inexcusable’ errors handling UK health data, says report
theverge.com
theverge.com
The amount of data obtained is frankly tiny compared to that available regularly to researchers in USA. E.g. as part of my PhD research I have access to de-indentifed data on 40 Million patients spanning 5 years from several states.
Not to forget programs like CMS Qualified Entity which provide access to identifiable data on ALL Medicare enrolees to companies. Such arrangements have been in place for decades. The only reason Deep Mind is being persecuted is since they are a juicy target, and fear of AI sells very well these days in academic circles.
The report does not lists any specific cases where violations occured but rather makes broad hand waving claims about cabining or advertising.
For those interested here is the orginal paper, instead of verge article.
https://link.springer.com/article/10.1007%2Fs12553-017-0179-...
If they really had substantive argument (that indicated real malice on part of Deep Mind) it would have been easily published in Lancet, BMJ, even New England Journal of Medicine. Instead the fact its published in some subject specific Journal should tell you about concreteness of their "findings".
Arrangements that have been in the place for decades in the US are not necessarily relevant to the UK.
The report essentially says Google owns Deep Mind and Google has advertising business, ipso facto, some magical standrard dreamt up by the authors were not met.
The reason for mentioning US agreements is that several nations, researchers and comapnies have developed procedures around sharing this data. And such sharing is not entirely unprecedented.
None of this is good for the individuals whose data is potentially at risk, and it's perfectly fair and reasonable to ask whether some of the most sensitive data there is about individuals is being properly handled by those entrusted with it given the implications of modern technology. There might be a lot of potential good in big data analysis for improving healthcare outcomes, but there is also a lot of potential harm if people stop, say, being confident in discussing potentially limiting conditions with their doctors, or reaching out for help with mental health issues because they're worried about confidentiality.
Regarding your second point, yes all these issues have been considered in depth by policymakers in US government and NHS. And current rules/contracts already incorporate legal protections to prevent misuse in form of both civil and criminal penalties.
If DeepMind had done something truly unusual or nefarious they would have been already prosecuted. But having failed to find any evidence of explicit misuse the authors of the original paper have shifted the goalpost into territory of vague arguments.
It does, though. Legal contracts aren't magical spells. Audit-ability is important when data sets are particularly sensitive and extensive. Which is why the new agreement includes provisions for auditing.
edit: NVM, I see down-thread what you mean.
Nope; contracts create enforceable expectations between parties but do not, except in exceptional cases where the law allows for this, override the requirements that exist in law.
2) There is not such thing as "anonymized data" that is gathered by companies. There are several studies out there that show that with only 4 data points you can pinpoint someone in an "anonymized database" with 90% accuracy. Even Apple's differential privacy can probably be reverse engineered to find people. Just because you don't care to do that, doesn't mean others wouldn't do that either.
More to the point, author almost certainly operated under a heavily and independently vetted IRB protocol that would have made such a purposeful re-identification a criminal or at least very serious civil offense.
Ultimately, the lack of a legally binding contractual obligation with sufficient audit-ability is primarily what Julia Powles & Hal Hodson are criticizing.
From the verge article:
"The data-sharing agreement — which was signed in 2015"
There was a legally binding contractual agreement between the two parties Deep Mind & Royal Free.
Here is the one that I and thousands (I am NOT exaggerating) of other researchers regularly sign to get access to data.
Which was, allegedly, far too lax.
The claim that DeepMind was providing "direct care" is particularly questionable IMO, and that has significant implications in the context of this agreement. It means anyone who slaps together an iPhone app (and has the right clout/deep pockets) can get access to full and fully identified medical records without patient consent. IMO that's the big story here, and it's troubling.
The "don't worry that's covered by existing law" responses were also problematic. Why does this particular contract need to depend upon the enforce-ability of that law? What happens if the law is repealed and DeepMind happened to keep a copy of the data? We want the protection here, in this context, so just put it in this agreement. Which is what they ultimately did in the new agreement AFAICT.
Also, the data shared is extremely broad -- I doubt you could get access to a data set of that size and quality without patient consent through standard channels, for example. I've never seen an identifiable data set containing "not only to relevant blood tests and diagnostics, but historical medical records dating back five years, including information on HIV diagnoses, drug overdoses, and abortions" for 1.6 million people. Access to such sensitive data creates a difference in kind that justifies greater care and skepticism, especially WRT audit-ability. The parties appear to agree, since they signed a new agreement recently!
Demonize these researchers all you want, but as a direct result of their work, NHS patients have stronger legal protections today than they did a year ago.
Ummm thats how the world works, frankly its like saying water is wet. Whenever a hospital works with any third party provider, it signs a BAA agreement which governs the sharing of data. All risks of disclosure, etc. are priced into the contract. Any large hospital will have dedicated group of lawyers for sole purpose of drawing up these agreements.
>> NHS patients have stronger legal protections today than they did a year ago.
No they don't. For starters Deep Mind has not even returned any collected data.
Most BAA agreements are for things that are actually and unambiguously direct care, and for the actual patient being treated, and only obviously relevant data used for a particular business processes.
Like lab results or billing, for example.
Most of the data Google received was for people who it's unambiguously NOT providing direct care to. And even for the people who might be helped with their app, IMO "direct care" is still a stretch.
So, this agreement was not standard or normal in terms of scope or quantity. That's almost tautologically true, since what DeepMind is doing is presently abnormal.
Now, you may argue that it should be allowed without additional data protections. But don't mis-characterize. Most BAA's are significantly different in scope and purpose from from this agreement in any number of ways.
> No they don't. For starters Deep Mind has not even returned any collected data.
They've agreed to independent auditing, which is a huge improvement. Source: https://deepmind.com/blog/working-nhs-build-lifesaving-techn...
Personally identifiable health-related information is classified as sensitive personal data under the DPA, and as such there are particularly strict conditions on processing it. What two organisations write in a contract does not change this.
http://searchhealthit.techtarget.com/definition/HIPAA-busine...
The laws are very different, and there are much stronger protections in the UK.
The items raised with DeepMind would raise serious concerns under HIPAA of the entities involved were covered by it.
2. LOL yeah you are so damn clueless that I wont even bother replying. Lets just say there is reading some popular press article about privacy and annonymity. And then there is knowing intricate details of how business/research is conducted.
Then go ahead and try to get access to a similar data set. Unaudited access to fully identified and full medical records going back a half decade for 1.6 million people, without obtaining individual permission or providing any individual notice. For nothing other than the promise of some diagnostic assistance software. As a for-profit company.
Good luck greasing those wheels ;-)
It also appears that you are casually dismissive of concerns about the risks of that data being de-anonymised. Your best rebuttal seems to be some vague allusion that the parent poster, who was essentially correct, didn't know what they were talking about. (This is something of a digression anyway, because in the actual case we're talking about, as the report notes, the data supplied was already identifiable anyway.)
You have accused the authors of the report of going on a witch-hunt, but again you have offered little real argument except that such things are going on in other places, as if that makes those practices above criticism or automatically acceptable.
Do you realise that you are providing a near perfect, real-time example of the need for stricter controls on medical data and who is allowed to access it?
And regarding your concern about me, if anything I should be the one engaging in this ridiculous witch hunt against Deep Mind since I have during my PhD developed an Open Source transparent analytics platform for data on millions of patients.
But unlike the authors I want real debate and real systems that can be used, not faux outrage over another click bait article.
That still doesn't matter. If Royal Free shared personal health data with Deep Mind without the data subjects' explicit consent, and if the arrangement doesn't fall under other permitted conditions for processing sensitive personal data under the DPA, then someone is breaking the law. If there were not at least some legitimate grounds for concern about whether that is the case, it is unlikely that the ICO would have opened a formal investigation into the deal. Again, nothing written in any contract between the organisations involved supersedes our primary data protection legislation. As for standard processes for handling such situations developed by government agencies, another substantial part of the report was about how the various regulatory bodies do not seem to have been consulted in ways that would normally have been expected before starting a project of this nature.
There is nothing routine, in the UK environment, about transferring complete, multi-year medical records for large numbers of identifiable individuals to external organisations not run by medical professionals for such open-ended purposes as those described in the agreement here. This is not a small thing.
Substantively improving patient privacy protections in a concrete case by forcing a large corporation to agree to strong privacy protections and auditing regimes seems like a "real" contribution spurring a "real" debate.
For 1.6 million people, the results stemming from this paper much more "real" than any number of git commits to a software system.
hahhahah.
Chill dude, chill.
Software changes the world.
Well, it's true. Differential privacy is a nice idea, but in this particular case, you're disparaging a style of research that -- to date -- has had a much greater impact on improving actual, real world privacy than all the fanciest query engines in the world.
> Software changes the world.
The most important thing to know about differential privacy is that when in comes to privacy, software always plays second fiddle to policy and politics.
Differential privacy algorithms are literally nothing other than the implementation of a legal spec. Without the law, the algorithms are pointless.
Mind you, I don't intend to disparage differential privacy work in any way! But disparaging policy research while holding up differential privacy systems as the answer massively misses the point...
I actually don't use differential privacy at all!
My software implements legal requirements as stated by the agency providing the data. Which are equivalent to a stronger version of K-Anonymity.
I am not at all against Policy Research, the issue is that this particular paper is a spectacularly BAD example of Policy Research. Policy research should not be driven by FUD around AI (e.g. the quote "did not constrain the company from using AI analytical techniques on the data") or some corporation.
I was just going by your own description on the product website. Anyways, kind of irrelevant since your "software implements legal requirements as stated by the agency providing the data". That's basically my whole point.
> ...is a spectacularly BAD example of Policy Research
I tend to judge research by its merit and its impact.
Merit is discussed at length elsewhere, and IMO you're flat out wrong about the normalcy of this particular agreement. But we can leave that to other threads.
In this thread, you're disparaging the research based upon its impact ("real debate", "real systems"), when in fact it has had a direct impact.
I dunno; if your cavalier attitude toward privacy protections around personal health data reflect how your research was managed, there could be a very strong personal-interest reason for you to be waving around the "no big deal" flag you are waving about DeepMind.
Even the expert from Wellcome Trust thinks its “overly-critical”.
Hello. I am one of the authors. We are on the record saying that we want a real debate.
See: http://uk.businessinsider.com/deepmind-royal-free-london-nhs... https://www.theregister.co.uk/2017/03/16/googles_deepmind_an... http://www.bbc.co.uk/news/technology-39301901
Do you mean personal or political? Because you insinuate personal motive with no evidence ("...and fear of AI sells very well in these types of academic circles...").
However, I don't see anything particularly wrong with researchers being motivated by politics.
Personal privacy is political.
Climate change is political.
Human rights are political.
Researchers should not feel an obligation to shy away from politically charged topics. At least, not if we want science to be anything except a tool for moneyed interests to batter the rest of us into submission.
Shaming researchers for publishing politically relevant findings is its own form of witch-hunt.
> E.g. as part of my PhD research I have access to de-indentifed data on 40 Million patients spanning 5 years from several states.
De-identified is a rather extraordinary caveat! It's the difference between "aub3bhat has cancer/AIDs/depression" and "entry 143048 has cancer/AIDs/depression". Or, for that matter, "aub3bhat says..." and "<insert your name> says..."
In short, the situations are nothing alike.
> The report does not lists any specific cases where violations occured but rather makes broad hand waving claims about cabining or advertising.
If companies are willing to open-source their analysis tools and provide a mechanism for me to audit how my data is used, then I'm happy to not assume malice.+
Otherwise, I'll assume that they do everything that's legally permissible.
Absence of sufficient contractual protections is enough justification for deep concern, IMO.
IANAL, but I imagine you'd be hard-pressed to find a competent lawyer giving a corporate client literally any other piece of advice. For example.
> The only reason Deep Mind is being persecuted is since they are a juicy target
You say this as if it's unreasonable. Google's very explicit raison d'etre is to collect and organize the world's information for profit. Explain to me again why I should trust them with my health records.
--
+ To DeepMind's credit, the article indicates that they're planning on doing something like this, and an improved contract has already been signed. And I don't fault Google or DeepMind in any of this. The fault is entirely with NHS.
Finally their is nothing wrong in writing a paper about a politically charged topic, but completely dismissing existing standards to paint a picture of doom and gloom is ridiculous. E.g. just look at the references cited by that paper.
From what I understand, this program has exactly the sort of contractual legal limits on use that the authors are asking for:
"Qualified entities may use the information obtained under section 1874(e) of the Act for the sole purpose of evaluating the performance of providers of services and suppliers, and to generate specified public reports".
> but completely dismissing existing standards to paint a picture of doom and gloom is ridiculous
The authors don't dismiss existing standards, but rather explain how the DeepMind agreement was particularly troubling in light of standard data sharing agreements.
And in any case, two wrongs don't make a right. If NHS hands health data over to private companies, there should be very strong legal protections governing the use of that data. PR statements and unenforceable promises are not enough.
To the extent that it is common for private companies to get access to identifiable health information without agreeing to strong legal protections, the authors of this paper are completely validated re: their doom and gloom tone.
And to the extent that this is not the case, the DeepMind agreement was exceptional and particularly troubling.
In either case, you're just strengthening the paper's thesis.
NHS hands health data over to lots of private companies. Like any corporate entity, NHS would be expected to perform better when it outsources non-core work. The core work of the hospitals is patient care. The core work of the health care system is coordination, neither of which sounds like "algorithm development".
Keep in mind there is a vast standing body of law and precedent around these issues in the US, Britain, and anywhere else that medical research is undertaken.
This is not how this works. You can't really ensure anonymity with data like this.
Here is more information about the dataset, they themselves are aware of it. https://www.hcup-us.ahrq.gov/tech_assist/tutorials.jsp
In fact one of the major goals of my research is building an online plaform that ensures such large datasets no longer have to be distributed, as they are today.
> 1) We do not know––and have no power to find out––what Google and DeepMind are really doing with NHS patient data, nor the extent of Royal Free’s meaningful control over what Google and DeepMind are doing;
> 2) Any assurances about use of the dataset come from public relations statements, rather than independent oversight or legally binding documents;
> 3) The amount of data transferred is far in excess of the requirements of those publicly stated needs, but not in excess of the information sharing agreement and broader memorandum of understanding governing the deal, both of which were kept private for many months;
> 4) The data transfer was done without consulting relevant regulatory bodies, with only one superficial assessment of server security, combined with a post-hoc and inadequate privacy impact assessment;
> 5) None of the millions of identified individuals in the dataset were either informed of the impending transfer to DeepMind, nor asked for their consent;
> 6) The transfer relies on an argument that DeepMind is in a “direct care” relationship with each patient that has been admitted to Royal Free constituent hospitals, even though DeepMind is developing an app that will only conceivably be used in the treatment of one sixth of those individuals; and
> 7) More than 12 months into the deal being made, no regulator had issued any comment or pushback.
Quite a few of these strike me as rather absurd, but I don't know the regulatory environment in the UK.
In a "that can't possibly be true" sense? Well, yeah, that's kind of the point...
1-3 seem like the sorts of things that even the least privacy-sensitive person can agree are troublesome.
If Google is willing to give anyone who signs a set of modest legal agreements carte blanc unaudited access to data stored on their servers, I'll begin to even remotely consider entertaining the claim that 1-3 aren't important.
5 in particular is blatantly illegal in the UK unless DeepMind is providing direct care. They claim apps == care (IMO absurd).
6 should just straight up be illegal.
3. If you want to learn new insights, you — by definition — need to include data that a priori don't seem directly related. This point even notes that the data was technically and legally well-scoped.
5. Sounds like they were within the terms of the existing data privacy agreements given 6. Again, I don't know UK privacy laws.
6. My impression from the paper is that they're not only trying to manage AKI but also improve the detection of it. Ok, sure, they're not going to improve detection in deceased or transferred patients. Those probably should have been minimized.
I'm not opposed to that in general, but there really ought to be 1) an opt-in or at least a well-advertised opt-out mechanism; and 2) an independent audit of how data is used.
FWIW I think this was a healthy push-back against "just trust us" and hope the result is a cleaner template for future similar projects.
> The report notes that DeepMind was given access not only to relevant blood tests and diagnostics, but historical medical records dating back five years, including information on HIV diagnoses, drug overdoses, and abortions. The report also says the wording of the 2015 deal did not constrain the company from using AI analytical techniques on the data (something DeepMind disputes).
What's the legal status and overall vibe of something like this in the UK?
It's a bit different in the USA as we don't really have an NHS here with everybody's data and it'd be done directly with multiple insurers or medical providers. I'm guessing this would violate some type of patient privacy laws as well.
Given the option I bet many people, including your humble commenter, would opt-out too. (or just not opt-in if we're lucky).
> including your humble commenter, would opt-out too. (or just not opt-in if we're lucky).
At this moment, do you know what organizations own a copy of your medical records -- in whole or part -- and what laws apply to each of those partial records? If not, then how can you possibly hope to opt-out?
I don't and that's partly why I prefixed that line with, "Given the option ..."
The most basic issue is that health records about an identifiable individual are classified as "sensitive personal data" under the Data Protection Act, which is our primary privacy legislation. As such, there are several extra conditions that apply, in addition to all the ones covering all personal data, which constrain how "data controllers" and "data processors" are allowed to use that data. (The technical terms are defined in the Act itself.)
A lot of the details discussed in the original report are relevant because in this case the "data subjects" (the patients) did not give their explicit consent. There are specific conditions required under the Act for processing sensitive personal data without such consent, and it's not clear whether this agreement met them, for reasons such as those discussed in the report we're talking about. If in fact the conditions were not met then a lot of people have probably broken the law, and both the organisations involved and their officers are potentially guilty of offences.
We don't really have that in the UK.
This is one particular NHS Trust that runs 3 (I think) hospitals (and some other services) in London.
So if I go to them to have my knee replaced, and I go to a different trust to have a lump-ectomy, the data Deepmind gets doesn't include (I think) the lumpectomy.
As for your question about whether this kind of thing is popular: a while ago the NHS wanted to run something called Care.Data (care dot data) which was a big data project. A bunch of people complained about not being allowed to opt out, and so an opt out was added, and a bunch of people then opted out. Personally, I would have opted in if they'd given that option.
Still a very legitimate concern, especially in the age of privacy.
The data needs to be broad if you are interested in finding out things you don't already know... that's why it's being fed into a machine learning algorithm in the first place - If you get selective then it's not going to be very useful - how do you limit history to what's relevant when you don't know whats relevant?
There is however an interesting difference here between most data mining on the web which is trying to sell advertising. In this instance it should be fully anonymisable, only the doctor should be allowed to see that patient ID e76f57a is John Smith.
If you want to take a look at the underlying technology, take a look at https://github.com/google/trillian
Actually, the institution which collects and stores the data handed it over without due process. The article keeps trying to blame DeepMind, I guess criticising the NHS is a little stale. I think this is in a journal because it is too long for an op-ed but not substantial enough for a long form article.