2.5M Medical Records Leaked by AI Company
securethoughts.com
securethoughts.com
If I fucked up on this scale, jail would be the least of my worries. I'd never work again. I'd be professionally shunned forever, with no hope of redemption once my name is attached to the incident in some Google search. No more grants, no more collaborations, no nothing. I'd be ruined. Frankly I'd be begging to be let into prison; where else am I gonna be able to eat?
Nothing will happen to these people.
Secondly, this seems to be a straightforward database misconfiguration. The database was set to public with no restrictions. It likely wasn't accessible to anyone who wasn't explicitly looking for it, i.e. security researchers or black hats. Yes, this was horrible, but nothing bad happened here. The database was simply set to private.
Third, my friend, chillax a bit. Yes, it's extraordinarily important to protect people's medical data. But no, you're not personally liable if a fuckup happens. Everyone fucks up. That's why the fuckups are covered by legal protections. You take your job seriously. Good! But you also take your job way too seriously. Bad. You're gonna burn yourself out within a decade with this mindset.
None of this is to say that what happened didn't matter. Of course it matters. But you are literally saying you'd be begging to be let into prison. That's not proportionate, and feels like a reflection of the current social climate of retribution-as-forgiveness.
It isn't. I get that experience as a pentester can make you jaded to security practices, but the healthcare space is extremely regulated and much of what the grandparent said is a direct result of that.
> Secondly, this seems to be a straightforward database misconfiguration. The database was set to public with no restrictions. It likely wasn't accessible to anyone who wasn't explicitly looking for it, i.e. security researchers or black hats. Yes, this was horrible, but nothing bad happened here. The database was simply set to private.
This isn't just a minor misconfiguration. Again, this is a regulated space, with rules. There should be multiple layers in place to make sure that this is not opened to the public. There's risk assessments that needed to be carried out when the database was launched, and on a regular cadence afterwards. "It wasn't accessible to anyone who wasn't explicitly looking for it" isn't reassuring. Look at access logs on any device with a public IP address and it's clear that people are constantly looking.
> But no, you're not personally liable if a fuckup happens.
Under HIPAA, you can be held personally liable. This includes jail time for certain classes of violations.
> But you also take your job way too seriously. Bad.
People working in this space are required to have regular training and undergo audits of their activity, what information they access, risk assessments on how it is stored, and long, drawn out compliance activities just to get access.
Has anyone ever been actually sent to jail under HIPAA for something comparable to an honest DB config mistake? Has anyone been credibly threatened with it in the court of law? Has anyone come remotely close?
It's the same thing with PCI. They say you can go to jail for violations but exactly 0 people have ever been prosecuted, despite constant negligence in the payment industry.
Suppose you fucked up, and the worst happened: You were the reason the database was misconfigured. What then?
Then you live your life. You think future employers are going to pass on you because of it? Maybe some would. Most don't give a shit. They care about one thing and one thing only: whether you can help them achieve their objectives.
Morals are a fine thing, but the reality is that the original comment was correct: Nothing will happen to these people. As it should be.
They are required under law to notify the affected people. THat's the extent of what will happen. There may be legal repercussions for the company. And I'm sure the engineers will feel pretty awful at this mistake. I know I would. Everyone does.
But that doesn't change the fact that the world is insecure.
Yes, I'd be a bit surprised if this database hadn't already been detected by some automatic IP and port scanning scripts, and the wrong people found out about it
(I don't with exactly these things though do I might be mistaken)
Just to be explicit about it, I'm not suggesting the release of this data isn't a problem. Just that as medical data goes this is about an innocuous as it could have been. Even day-to-day visits to your primary care doc would probably have references to much more sensitive health data.
At least there's a cynical take on this. They have no inherent interest in protecting this data, absent external pressures or legislation that imposes penalties.
It strikes me the real issue is the lack of negative inventive for people and businesses who screw up with data. If you build something physical that falls down and hurts people, your investor money ends up paying compensation. In tech though, the goal seems to be to make the revenues without ever touching the responsibility.
The story over the weekend of Amazon trying to avoid being deemed liable as a seller for goods they ship, (often) deliver, warehouse, market, dispatch and receive seems to sum this up. A race to the bottom when it comes to protections - Amazon effectively is a shop selling goods from others, as it all happens on their terms. We see similar in physical retail where the manufacturer or reseller has a retention of title over goods in the store, and that doesn't affect the store being responsible. It seems like the "being responsible for what you build" is very much lacking in tech companies, due to lack of external regulation to achieve it. Same here in this case.
The user already pays for the insurance (or hardware or service in other cases), so the data collected should only be used for those purposes, e.g., handling the claim and not building AI models without the necessary care. The customer should have the right to sell their own data to a company they see fit to take care of it appropriately and that compensates them for it.
The reason you'll get yourself in trouble and these people won't is because you are operating in different moral systems. Where they operate, the moral thing to do is whatever it takes to succeed, it's entirely moral to "fake it until you make it" to buy media articles promoting your product, to sell media space your readers believe is actually curated, to the highest bidder.
You operate in the "ivory tower" they hate, because it's slow and puts up boundaries that limit people from succeeding at all cost, that makes you think about the consequences of your actions. They call it slow and inefficient, I call efficient in the long run.
But I know plenty academic researchers with a "move fast and break things" and "ignore all rules" attitude. In any highly-competitive system there will be those that gain an advantage by being amoral while everyone else follows the rules. Healthcare is probably especially vulnerable to this as the rules (and legitimate companies following them) has resulted in healthcare lagging far behind other industries in terms of new technologies.
-M. Tullius Cicero. De Officiis. Book II.
I sat down with the CEO and discussed our plans (to upload the data to our S3 buckets with ACLs) along with all our other security plans, pointed out the two of us were professionally and legally on the hook (her more than me, as I was not an executive and had limited liability), and what I was doing to protect the clinical data we had. We looked at our plans, and ultimatley concluded we could both sleep happily at night with what we were doing.
It was definitely a lot of work to maintain the ACLs and monitoring and checking to make sure we weren't leaking data, and a lot of education for our scientists. Even if something negative had happened, we would have been able to foresically detect it, and work with legal authorities with confidence we had done a good-faith, better-than-average practices implementation.
Not true. They may end up putting out a press release explaining how "Here at Company X, we take security very seriously. Securing data is very important, and we will investigate ourselves to understand if we did anything wrong." So yea, consequences!
I wonder what would happen if devs could say to their boss "no, this won't be released until we're confident in it" with legally enforced immunity to consequences.
I guess I'd argue that those people shouldn't be legally allowed near this kind of thing without that kind of a certification. Looking into all of the other engineering disciplines, that's exactly the kind of thing you see. I have a BSME, but I haven't taken the Fundamentals of Engineering exam to get my FE cert, in part because getting a PE certification requires working underneath a licensed PE for a certain number of years, which isn't the case for my current job.
I also know that by not doing so, there are certain projects that I simply can't work on. I have to imagine that there's a way to create a legally enforceable framework that falls into the same category for software engineers. Want to build a company that creates a digitally-synced notepad? Have at. Want to touch personally-identifiable medical data? Better have a licensed engineer working on that project to sign off, else your company is wide-open to liability claims with teeth. If something unreasonable gets by the signed-off engineer, they're on the hook too.
Obviously, it's a complicated problem, and reducing things to a first-order solution rarely is a catch-all, but there has to be some more professional/personal responsibility taken by the individuals building these systems, and a requirement of licensure is a way of empowering engineers in those positions to the point where it actually matters.
You answer your own question fairly well, but I'd add the observation that in licensed engineering domains, we don't always require licensed engineers. We have a licensing regime for structural engineers, but we don't require them for minor structures like gazebos or doghouses.
We could have licensed Software Engineers, but only require licensed oversight for software dealing with human lives (avionics, medical devices), PII, elections, and a few other critical cases.
I developed software for medical devices and you have to do a risk analysis, formalize the software development process, declare qualifications of people, make it revision proof, have a formal testing process, ... everything is already accounted for.
Notified bodies ensure compliance. They have the problem that they cannot really evaluate the work of software engineers of course. Not even another software engineer could do that within feasible time limits. No software engineer can make sure there aren't exploits that could endanger user data. You can at most test if due diligence was ensured.
The manufacturer is responsible for ensuring safe operations of devices and yes, that includes keeping personal data safe.
But again, the problem wasn't the engineer at all, the problem is the wish for amassing data like this. Paper license or not, it rarely ensures competency and wouldn't have solved this problem.
Aside from legislative issues that ensures that user data belongs to the user the data is about, ensuring that companies don't sell and share medical data with "friends and family", ... this is probably the last step, if it is even required at all, which I would dispute. There are no guarantees if you amass data like it was done here.
I've worked in critical infrastructure work and retain an interest in the field. A professional software engineering license should be legally required for certain classes of risk.
Ultimately the answer to an irresponsible feature ask should be:
> I will not approve implementing this feature, because in my professional training and experience, the risk exceeds the acceptable tolerances for a (spaceflight, medical, power systems) delivery. If I implement this and a failure occurs, I will be in court and my career over: I refuse.
When a bridge falls down, building collapses, patients die, people take notice.
Meanwhile, we've had the social insurance numbers and banking history of 165M+ UK, US, and Canadian people leak out of sheer technical negligence [0], and it resulted in a meek settlement and hardly broke through the public consciousness.
It seems to me that until someone dies in a way that is very clearly linked directly to a woefully negligent and under-trained software engineer messing up in a very public way, the needle is not going to budge at all.
Perhaps autonomous cars? Even then I doubt it, to be honest.
Today, it's well under that.
It is, of course, long overdue; Therac25 should have really gotten the effort going, but, ce la vie with a irresponsible economy. It's plausible the EU legal systems will develop this effort first. I would not be surprised to see France or Germany fully develop the idea, probably in connection with Airbus or Siemens. Anyway. Idealism around quality....
So is that the remedy people are recommending here?
If I remember history correctly, there wasn't one boiler explosion that caused us to start regulating who was allowed to design boilers. It's just that as boilers became more and more popular at the end of the 19th century, people started being maimed and killed more often, and at some point we just as a society decided enough was enough.
You can still effectively do exactly the same job as a licensed engineer without being licensed, you just can't sign off on work (And thus be held responsible).
The way I see it, creating a license for Software Engineers doesn't really fix the issue. It just creates a scapegoat to blame when things go wrong.
That's just how it works for other engineering disciplines; with the license, the scapegoat has the legal power to refuse responsibility until they're confident things won't go wrong.
Even top software companies have things break or go wrong very frequently compared to traditional engineering.
I don't see this as a good solution in the software world.
Hey can you put together this trillion piece jigsaw puzzle? Oh and we need you to sign off on it. You're sure it's not going to kill anyone right?
Works on my machine!
Then you say "no". Business "needs" are important for replaceable serfs with no power to say otherwise; when the business needs engineering sign-off, you tell them what they needs to do.
https://www.vice.com/en_us/article/yw798m/oregon-unconstitut...
Yeah, just what the software industry needs.
You are willing to pay the costs of "professionalizing" software development and installing expensive gatekeepers, certifications, signoffs, and processes at every step, right?
Most of that is already in play already after about 30 people are in a group, we as an industry haven't formalized it, and there's no legal teeth around it.
I don't see anything changing as long as it's in money's interest to stay where it is (money rules this country point blank). Right now it's in too many business's interest not to change. There are also slews of developers earning quite a bit who might be forced into career changes depending on how licensing could be implemented.
We spend about 15k per year for about 10mm in coverage (we are a small shop). This sort of oversight is not just an engineering one but it’s fundamental to the core ops of the business. If we are rushing under client pressure (or just running late) to the extent we take on risk to trigger liability, it’s full stop.
https://cense.ai/about Guy number two here had the background to know better. Just didn’t make it a priority. It’s unfortunate. The only way things change is when we start seeing data warehousing/collection as a liability (not an asset) and manage it accordingly. And making the penalty for error unforgivable.
The irony is, of course, that PL/E&O insurance for software work runs pretty cheap, presumably because liability on the part of developers of software is quite rare!
Why can medical records be sold for so much? What value do they provide to the buyers?
Can you really fake bill enough before getting caught to be worth burning an M.D.?
I never really understood this angle. Say the insurers use a phone call to verify 1% of treatments with the patient. Say 20% of those calls are answered and produce a coherent reply. That ought to be enough to make the expected payoff negative.
Either way, this problem isn't unique to the medical world. Every other industry has to prevent fraudulent invoicing.
https://www.google.com/amp/s/www.nydailynews.com/news/nation...
... which was pretty much my point.
If they can steal money out of doctors' bank accounts, or redirect it on its way in, they don't need patient data. The doctors already have patients. And payments coming in. This isn't like income tax refunds where the payer pays each individual.
In fact, if they have patient data, it's stupid to use it. What insurer isn't going to notice some doctor suddenly getting a thousand new patients all in the course of one week?
Sorry man, it still doesn't add up. If you're equipped to do this at all you're equipped to do it without needing patient data.
With nuclear material, the IAEA has set up a sophisticated surveillance regime to make sure that countries use nuclear technology for peaceful purposes. Everything that contains nuclear material is video surveilled. They collect probes and check composition to prevent someone taking, say 1%, replacing it with filler material, and then mixing it again.
If we concentrate data, we should employ such surveillance regimes for highly enriched sensitive data as well.
I'm working on a team writing an in-depth whitepaper on this topic right now, I'd love to link it. Perhaps when it's published.
The execs feel they need to show they at least made an effort, and these companies branding themselves AI are easily able to take advantage. They market their products as these complex decision-making/analytical systems, but most of them are either glorified reporting dashboards or just an integration point/pipeline builder for a set of services.
The business buying this stuff doesn't need any of it to become more efficient, they are effectively spending the money to be able to generate the hype of saying, "We're in the AI space".
"There are certain gaps [in HIPAA coverage], like if a person meets with an accident on a car; the health insurer as well as the auto insurer shall receive the medical bills of that person. The health insurer is responsible for protecting the health data of this person under HIPAA; the auto insurer on the other hand does not hold that duty."
[1] https://www.andrealopezv.com/new-age-security-solutions-to-p...
The amount of bullshit stated around HIPAA is ridiculous
> there were multiple references to an artificial intelligence company called Cense. The records were labeled as staging data and we can only speculate that this was a storage repository intended to hold the data temporarily while it is loaded into the AI Bot or Cense’s management system.
Increasing bug bounties doesn't happen when the executives do not have a culture of security in their company. That's a loss cost center for them. We need a different incentives that go beyond just rewarding hackers with bug bounties.
Insurance is a great aligner is financial incentives.
It's only a matter of time before the whole thing gets leaked out, not to mention that compiling such data in the first place ought to be none of the govt's business. Such data is simply too powerful to be entrusted into the hands of any single party, political or corporate.
> It's a collective action problem that needs collective action. Individual incentive changes aren't going to cut it. Unfortunately many thought leaders in the industry have tried to build a culture of suggesting that the solution to collective action problems is individual iteration.
Most kinds of compliance are linked to legal costs as the ultimate source of consequences, not so for PCI.
The ultimate costs for failing to comply with PCI are the actual costs of card fraud which don’t depend on anything in the legal system. When your regulations are designed and enforced by the entity that actually loses money when they aren’t followed, motivation lines up and they work better.
Medical data is... not that
If HIPAA could pierce the corporate veil, this could no longer occur. Wanna dick around and not do your job as a founder / investor? Your personal assets are on the line.
To stay safe with medical data, however, you basically just need to hit whatever standard you think is reasonable. There's no established standards other than:
1. "PII" encrypted during storage/transfer.
2. Customers can request a download of their data.
3. Customers can request you delete ALL their data.
4. Fast track sec fixes above all other company goals.When this doesn’t happen right there are penalties and notifications. If correct, a breach of this magnitude is entirely possibly a death penalty for the businesses involved, and criminal liability.
If it isn’t, then it isn’t the business sector being unprepared or the regulations being missing but the legal system failing to follow through.
A small chance of a unilaterally set arbitrarily priced reward is being continually curb stomped by a two sided marketplace.
I think if there are at least social consequences some managers will start taking these stuff more seriously.
Just asking, probably it is not even possible with all the blanket "right to be forgotten" laws.
Here is a simple way to do this: https://yourdigitalrights.org/d/cense.ai (disclaimer I'm the co-founder of this free, nonprofit service).