Hospitals are a weak spot in U.S. cybersecurity | Hacker News Reader

Hospitals are a weak spot in U.S. cybersecurity

axios.com

242 pointsswedtrue161 comments

Healthcare CIO here. This is true. Healthcare is still using paper fax. It has a 30 year old data interchange format that no one really supports because it's more profitable to lock in customers to your EMR. Healthcare is HORRIBLE about upgrading anything, at changing processes, and technological progress in general. Healthcare is VERY backwards from a tech standpoint.

Another problem is that EVERYTHING is custom, we use very, very few off the shelf solutions. Need an EMR? Let's build it in MUMPS, a 51 year old language that originated on the PDP7 and call it a state of the art system like Epic or GE Healthcare. Don't like the terminal interface? Let's slap a GUI on the front that still interacts via TTY on the back end. SQL? Nah. C, C++, or any more modern language with more robust features and way more programmers? Nope.

Now, there are some EMRs and other healthcare-centric apps that are better written, but they're also terrible. Healthcare is a relatively small market, you'll never sell a million units of your app, so you charge out the wazoo for it, get a few health systems on it, and allow they to go crazy with customization to help lock them in. And then you try to add on modern security features on to a system that's been growing for 50 years and it's a nightmare. It's INCREDIBLY common for nurses and doctors to need to have administrator access on their Windows desktops for various apps.

I was about to leave IT in general when a healthcare gig landed on me, and I'm glad it did. I find it very refreshing to be in an industry where it's so far behind that there are mountains of problems to tackle, even if half of them are so stupid it makes me want to cry.

Regarding legacy EMRs, are specifications/standards like HL7's FHIR actually gaining any traction and making data interoperability more feasible?

Not OP but I can tell you that due to the object size limitations of FHIR objects, document exchange is just not feasible. Using FHIR to register a patient is pretty pointless currently as every older and larger hospital sends HL7v2.

RHIOs building gateways for 3rd party apps is very much the future of FHIR. But they’ll still be interacting with that crufty legacy system.

Just my own two cents.

Thanks! I was not aware of the object size limitations nor of RHIOs

Are RHIOs using engineers/consultants to wrapped legacy systems then devise methods to make accessing that data easier -- how are RHIOs using FHIR and legacy system to construct the future of medical data interoperability?

RHIOs exist predominantly to allow the exchange of patient information between participating hospitals or even regions. I personally only have insight into how two RHIOs operate. Both are very interested how they can turn FHIR into another avenue to distribute patient information.

The issue is that most EHR/EMR vendors have very limited limited interoperability with FHIR. As a lot of these vendors are struggling with CCD 2.1 implementations. The exciting space is allowing patients through 3rd party apps to request specific information or even send it to the HIE (RHIO in this case) and let there GPs know of certain events.

Which means a lot of work to bend over backwards to get APIs exposed for these 3rd parties.

In my world they sure are. Want health records on your iPhone? Well that comes via FHIR. You can even see the FHIR resource JSON in the Health app.

But there are many systems in a hospital. And as EuphoricEmu pointed out within the hospital, admits, discharges and movements throughout the hospital are still done via HL7v2 (a delimited and structured format).

Additionally, I would absolutely NOT build a new system on HL7v2 at this point in time. I would only use it to integrate with existing systems.

Also, I do know of EHR systems that use FHIR for their internal data storage format.

Unfortunately consumer-facing use cases are the ones big EMR vendors like Epic are focusing on with FHIR, push is often neglected or left out meaning we still have to rely on good old HL7v2 to get real time feeds out of the system.

Yes exactly! Since patient movement within a hospital is done primary with v2 and then hours or even days later a patient summary CCD might get sent across. Unless you’re dealing with encounter based CCDs. But those still aren’t live.

FHIR Subscriptions do exist.

Yup, there are a few EHR vendors that have some limited support. Intersystems’ whole platform is shifting towards have pure native support for FHIR. It’s still very rough and to into the weeds of things you sadly have to delve into that very old language (objectscript = MUMPS). But thankfully there bindings for more modern languages for it.

I work as a company that builds software to optimize dosing, and we have apps in a few of the major EHR vendors app stores. FHIR has been great for getting and presenting data - although I think true interop between EHRs using FHIR would be challenging, due to the large amounts of data and the (relative) complexity of each EHR. Despite everyone using FHIR, it does suffer from the same problem other specs like HTML do, everyone has an ever so slightly different implementation, and once it's been implemented, no interest in changing or improving it.

Could not agree more! As someone who works as a contractor for two relatively large RHIOs I was absolutely dumbfounded by how contradictory everything is when I first started. I’m in a unique position where I get to interact with dozens of EMR/EHR vendors. They’re all terrible.

Cerner? Epic? Suck and suck!

People need to stop hating on fax. Hospitals still use fax because it is a much more punishable crime to tap phone lines which requires physical access, as opposed to a server that could be infected from a hacker halfway across the world.

I see you point in theory, but as a victim of identity theft, from what I can see almost none of these are enforced punishments. From what I experienced and from what I've seen friends experience in the US: - Someone can steal money in your bank account - Sign up for an expensive phone plan and get a $1000 iPhone upfront - Use your healthcare benefits

And you basically file a police report, hardly anyone cares, and you are left with a bunch of paperwork to go fix it yourself. You get the money back eventually with plenty of paperwork.

At which point do thieves committing all sorts of punishable crimes actually get punished? In my case, the person signed up for a line of credit at Lowes, purchased 20k of construction goods, presumable all in a videotaped store and got off scott free.

it has nothing to do with security. Nurses like fax machines because it gets them a break. I've seen them print e-refferals just to fax to each other.

That's not even the same thing.

People mess around with just about anything they can get their hands on- so what if the nurses send messages to each other and have fun? I'd be willing to give the people who take care of me a ream of paper if it meant they were in a good mood.

It has nothing to do with fun or messing around. It is about people being incentivized to avoid the tooling. I worked in US and Canadian health IT, my perspective is completely different from patient's. This may not have huge impact in high-end American hospitals, but in Canada where hospitals are underfunded and drowning in beuracracy it is a disaster

I’m willing to bet that most digital PBXs out there could be infected by a hacker from halfway across the world too.

Saw that happen yesterday. A vendor had insecure remote access setup to an older NEC PBX. Someone attacked it and was making international calls with it.

Fax is odd, it was a fantastic thing when it first came about, and it has some desirable properties.

- It's direct point to point communication (over a network)

- The transport network is dedicated and not open to anyone and covered by quite strong laws in many countries

- It's easy to see the history of communications

- It's easy to see if the other end successfully received something

- It's relatively standardized and ubiquitous ( in health )

Email would be the closest thing, but it doesn't have all the advantages, and the extra add ons that would make it better (like encryption, delivery receipt, digital signatures) are not standardized and/or ubiquitous ( and often hotly argued about )

So fax is the lowest common denominator, that, if it was proposed today, would not be accepted for many of its disadvantages, but it's now hard to find a way to replace it.

- It's easy to see if the other end successfully received something

I think this is a biggie. It means your workflow doesn't need to include going back later and checking to see if your document was received, and then trying to send it some different way. You don't have to guess which way the recipient is capable of receiving a message.

It's the original e-mail. ;-)

somewhat ironically, the fax will be captured in digital form, some "middle" person will read it, then work out who it is for, then email it to those concerned / attach it to a patient record.

Except seeing it was digitally received is often quite insufficient to seeing it was received by a human it was intended for. All too often in dealing with healthcare and gov't orgs our faxes get lost with no way of identifying where they went. Presumably it is a mismanaged shared fax inbox where individuals are not actually being alerted to their messages...

Some points seem contradictory. How do faxes have history that's easy to see, and if the other end successfully received, but it doesn't have delivery receipt like email?

not sure I'm understanding? But fax sends data via a protocol, it knows it has sent by virtue of the protocol completing. The fax can keep trying sending and it will keep track of those faxs that have gone through vs those that haven't. Email doesn't have delivery receipts....it's either done by bolting it on in clients or various hacks used by spamme....errr..email marketing systems.

Just because you get a receipt from the protocol doesn't mean it made it from the fax machine to the intended recipient. Folks who send faxes still regularly follow up with calls and emails, "just sent the fax did you get it?" and the answer is often "no, what fax?"

>- The transport network is dedicated and not open to anyone and covered by quite strong laws in many countries

is it? what if the hospital is using a VOIP solution?

...when it first came about...

All these issues could easily be resolved on the internet if someone bothered a bit. Keep a whitelist of connections, private-public key communication (you can exchange keys off internet if needed), receive and read confirmation etc. It's not internet's fault that some idiot is storing passwords in plaintext and/or sending them over unencrypted connections

No quite, many fax system are just modems, image conversion to pdf, email. Plenty to go wrong. Paper faxes might be revived by the machine by you have no idea who read it, or who didnt. Fax machine are typically MFP devices, so now you may have some part of a medical record on your photocopier HDD.

Many fax machine can be programmed over the wire, so maybe you have default pins and now your faxes are being forwarded and you don't know.

> It's easy to see if the other end successfully received something

This was not the case when I worked in healthcare. It was a constant back and fourth of "did you receive it?" over the phone.

Fax machines are just as insecure as that server. Last year taking over a network using just a fax number was demonstrated:

https://research.checkpoint.com/sending-fax-back-to-the-dark...

And because it's explicitly grandfathered in to HIPAA as a "secure" method of transmitting patient data.

Also, fax machines are very often just as internet connected as anything else. Email to fax, fax to email, fax-over-IP, it's not just modems dialing each other on copper anymore.

No Hospitals use fax because it's too hard to change, most information traveling through fax is sent via automated fax servers so it is the worst of both worlds, hack-able server and unencrypted transmission protocol.

> Hospitals still use fax because it is a much more punishable crime to tap phone lines which requires physical access

Punishable, sure, but that's CYA thinking. It's less secure, because there's no way to encrypt fax like you can encrypt email. Punishment doesn't help anyone except the CEOs, unless, of course, it was the CEO's information that got leaked.

Also, yes, phone calls are sent over the Internet just like emails. The big difference is, yes, that phone audio isn't encrypted.

That is not why they use them. Doctors like scribbling, and hate being told what to do, that's why.

Hospitals use fax because A) they think it's secure (it's not) and B) they think it's easier (it isn't)

I involved in building 2 systems for healthcare:

Patient privacy monitoring (ensure no one peeks into patient records inappropriately) and medication analytics platform (monitors suspicious and anomalous activity related to drugs and opioids diversion).

This of course deals with multitude of different systems, EMR's, data formats, legacy-everything etc, but the GREAT about this whole thing - is opportunity.

No one really tackled a challenge of visibility across different systems and data formats in Healthcare and it's lots of fun to be part of that.

("Fun" is non-corporate word - but it still fun!)

re: privacy monitoring

I prototyped an access logger based on tamper evident logs, which used rolling hash codes. Precursor to this blockchain mania.

re: visibility

I'm still bullish on the Translucent Database thesis. TLDR: Use the password salt + hash technique to encrypt data at rest at the record level.

Source: Designed, implemented, supported 5 regional healthcare exchanges. eg Brooklyn Health Information Exchange (BHIX): https://www.itnonline.com/content/medplus-implement-clinical...

Don’t forget the turnkey Siemens PET scanner complete with virus infested control PCs.

Sounds like local and state government.

How much end to end efficiency do you think a proper/average IT healthcare system would bring ?

The amount of complexity that it would be necessary to simplify and approximate would make any answer to this question meaningless.

And it's not only an IT systems problem. It's a comprehensive systems problem.

Which includes training, and counterparty expectations, and manual data entry, etc.

I'd wish hard to have a peek in these projects.

One of the reasons change is so slow in the industry is because there are many must-be-coordinated changes, with various independent parties.

E.g. if I update my system, you need to update your system

From what I saw, one of the biggest motivators would be carving out legal protections for trialing some smaller % of total workflow under new systems.

E.g. If you moved < 5% of your claims handling to a new automated system, and it started rejecting claims that there was an informal understanding would be patched up on the insurance side, then nobody could be sued via intra-party contracts (but still beholden to national / federal guidelines, of course)

By decreasing that first burden of migration, you might get more traction in aggressive IT updates.

> E.g. if I update my system, you need to update your system

I see you’ve played InternetExplorer X vs X+1 before.

> How much end to end efficiency do you think a proper/average IT healthcare system would bring ?

I have a long history in proper IT, and I'm very legally/regulationally knowledgeable, and in my two healthcare gigs I've made friends of the medical staff for improving responsiveness in IT and making things easier to use, while also reducing security problems by having both of those worlds of knowledge. Usually top IT management isn't technically knowledgeable, and frequently they're not even that good with regulatory knowledge. That makes it hard on the rank and file to be efficient. Not to pay my own back too much, but being well versed in both regs and tech helps a LOT in user satisfaction.

What were you able to make easier to use?

There are new data exchange protocols/formats which help to exchange between parties. They have been defined 2000+ so they are quite modern.

The only problem is that is such a diverse way the healthcare providers are implementing it. So you end up having provider specific code :)

True.

My ETL work in healthcare finally pushed me to treat the problem as screen scrapping. Bypassing all the attempts at formality, eg XSD.

The next step, which I prototyped but was put into purgatory by being acquired, was to simply capture all the data and use text retrieval tools, eg Lucene. Versus ingesting the data, normalizing it (to some schema) and populating database(s). Basically, postponing the translation/transformation of client data until viewing. Because, as you hinted, everyone does stuff differently, and clients generally have no idea what their data looks like until we showed them.

The proper solution would be to have direct access to source data, versus data feeds, but that ain't gonna happen until we have single payer, because the current incentive structure strongly discourages such simplicity.

Does Epic use MUMPS? I know a lot of professional nurses and the rancor around Epic is off the charts.

Epic uses MUMPS, but a nurse would not typically be interacting with that side of it...

As it should be. Why should companies brag about using SQL or Hadoop?

Yes, as does Meditech. Both are horrible products to integrate with and they’re rather common in large hospital systems.

Backend is all MUMPS. Frontend was for a long time coded in Visual Basic 6.

VB6/MUMPS stack is... not ergonomic to code in.

Epic is easy to hate (it's everywhere), and for good reason. However, the alternatives are not obviously better unless there's been some radical innovation. There are definitely systems designed for a particular piece of a hospital (ex, ER, or labs, etc) that are probably better than Epic is, but when it comes to having one system for the entire hospital, they're all pretty bad.

The main problem is that the customer is not the nurses, it's the legal/financial/administrative side.

> The main problem is that the customer is not the nurses, it's the legal/financial/administrative side.

This. The reason my medical staff like me is that I fight for usability for them, and push back against legal when they make requests that aren't backed up by the regulations. Legal hates me for the same reason, I know the regs and I'm willing to fight them on it.

Happen to have an email I could send you? It's sigint security stuff you might want to look into.

My email is jwconway at protonmail dot com.

Did anyone think to make languages that compile to those? I imagine that it would make an incredible amount of sense for Epic, at its scale, to write the equivalent of Typescript to reduce errors and improve productivity. Or do they just not have any sort of dev tools/research department?

They built their MUMPS IDE themselves, and their change tracking / bug management system. And effectively the database code is also entirely written by Epic- the business logic and database are both written in MUMPS and live side-by-side.

So they have quite a lot on their plate when it comes to tooling. When I was there, there was a truly horrible attempt at a DSL for code-generation that would build database querying code, but I found it both unwritable and unreadable.

> VB6/MUMPS stack

That sounds like a medical condition.

And after looking into the MUMPS language, I can see why they named it after a disease!

>Epic is easy to hate (it's everywhere), and for good reason. However, the alternatives are not obviously better unless there's been some radical innovation. There are definitely systems designed for a particular piece of a hospital (ex, ER, or labs, etc) that are probably better than Epic is, but when it comes to having one system for the entire hospital, they're all pretty bad.

Yeah, our org is on Cerner and there's excited talk around going to Epic in the future. People don't seem to understand that it's just going to be more of same in dealing with an old, monolithic system.

It does. The language predates no-sql though it hasn't been kept as up to date as C. It's a database language.

I've never understood hating on mumps. 66 vs 72 (c language) It's just shitting on the language for being old without a way to impact it.

It's so much more niche than something more widely used like C that it makes me very suspect.

> Healthcare is VERY backwards from a tech standpoint.

This strikes me as the SAP problem.

Everybody hated their big company's <foo> system. Until SAP came along and somehow made it orders of magnitude worse.

It's what happens when purchasing decisions are driven from the top. Sadly, in healthcare, given all the regulations, I'm not sure there is another option.

From my perpective, part of the problem is that any system is chosen based on political connections and/or bribes. And once any system is in place, there is zero reason to replace it until someone else puts down enough bribe money to convince a poltician to make another move.

Government promises, and thus government kills. A monopoly enforced by guns and prisons.

> Don't like the terminal interface? Let's slap a GUI on the front that still interacts via TTY on the back end.

This made me smile, because this is exactly what I did when working for a major health insurer...

What if there was an open source EMR? *nix backend with browser interface. The huge amount of money saved could pay for support, customisation and implementation. Perhaps I’m being naive, but a system where the institution has more ownership would go down well I think. We are about to get an Epic EMR and all the clinical staff are basically expecting a catastrophe.

The VA's old system (VistA) is that open source solution - but other hospitals didn't want to use or buy it.

I think the EMR industry is driven more by safety, liability, and revenue for hospitals (in that order) than by patient/physician desired features or security.

It's also difficult to build tooling that interconnects across all of the medical specialties - and with the amount of customization that some providers want.

Hey thanks, I hadn't heard about it [1]. Apparently VistA has the highest user satisfaction of any EHR! Sounds like what is missing is the equivalent of Red Hat, which can support and implement the product, although implementation would be much more involved.

[1] https://en.wikipedia.org/wiki/VistA

You could look at this from a couple of angles. From the healthcare business side, it's controlling costs, plus either regulatory compliance or revenue support (e.g., through provider lock-in). From the tech side, it's whether something developed 'in house' can be 'sold' to the private market: the economics about government institutions 'crowding out' private ventures argues that publicly funded innovations should not deny private profit opportunities. In the public sector this is controlled with ethics regulations that prevent organizations (and individuals) from benefiting from that effort: they legally cannot market or sell that work _at all_. With an open-source system, it's (business) risk management and (provider) lock-in concerns. Proprietary systems are used to satisfy both. Safety risks arising from cybersecurity concerns are almost always chained to something already managed by existing processes with an insurer at the end of the line.

Is VistA usable by sole practitioners?

Like GNU Health?

- https://www.gnuhealth.org/#/download/projects

- https://savannah.gnu.org/projects/health

- https://en.wikibooks.org/wiki/GNU_Health

Can't tell if it's any good, though I hear it's been used in some medical facilities around the world.

Let's build it in MUMPS, a 51 year old language that originated on the PDP7

It’s funny. You got Ruby guys out here who think every problem can be solved with a new DSL, and you guys actually have a DSL and want to get rid of it! Maybe the grass is always greener on the other side?

Nah. C

Also traces its roots to the PDP7 https://en.wikipedia.org/wiki/C_(programming_language)#Early...

Despite the awesomeness of our tech stack and execution, our customers (hospitals) were completely fleeced by our sales and project management teams.

During the mid aughts, hospitals simply didn't have the experience to defend against predatory consultancies.

Hopefully that situation has improved with the addition of people like you.

>we use very, very few off the shelf solutions.

Even off the shelf is no solution really. Everything is proprietary, you get locked in and years later the vendor isn't maintaining shit and moving away becomes an blackhole of a cost sink.

>It's INCREDIBLY common for nurses and doctors to need to have administrator access on their Windows desktops for various apps.

Honestly, needing admin on a "user network" device isn't the worst. You can still run malware that attacks the network via non-admin context :/ The best move is to use AppLocker if it's critical.

So true. Also healthcare software companies are shipping shit quality products and charging a fortune. Hospital CEOs are buying from mates. Fancy pants doctors are demanding exceptions. Shift staff are sharing passwords. Medical systems are not even capable of automating proper user access controls. The volume of data, especially medical imaging is growing at a crazy rate. Network links are under invested in. They sweat the assets to the points of lunacy. Ok I'm gonna stop.

Pretty much sums up my experience so far in healthcare IT, esp. the C-level and doctor part.

It's really tough. You have a function which is viewed purely as a cost center; you have a totally porous environment where you're required to admit tons of minimally-verified people into confidential spaces; staff and affiliates need different levels of access from all over the world; there are critical availability demands where temporary denial of service for security reasons is unacceptable; device development is optimized for safety and fault tolerance as opposed to security which isn't ever really tested for; patients need to be able to submit tons of data in myriad forms; there are few central clearing houses for transmitting data so people are all calling each other with minimal validation; etc

Oh, and you're ultimately sourcing truth from people who are minimally trained on (and have minimal time for training on) the system.

Because they've spent the last couple decades focused on medical training.

And patients that lie / dirty input.

Sure, use cousin x’s coverage. Nobody will freak out when your blood type doesn’t match the records...

It seems that hospitals are overly focused on bullshit security frameworks and box-checking, i.e., HITRUST, which in my experience results in many dollars going to consultants with essentially zero tangible improvement in information security. Worse yet, the false sense of security within these hospitals due to having a HITRUST audit report with a bunch of meaninglessness check marks prevents them from actually doing the work of securing information properly. Have worked in health-tech for a number of years.

On the flip side, I’ve long preached that compliance is not security. HITRUST CSF is a huge improvement over the previous state of healthcare IT, because HIPAA is not prescriptive

The famous critique on HITRUST by a healthcare security guy that went viral, calling it "Cumbersome, Expensive, and Arbitrary":

https://www.linkedin.com/pulse/open-letter-hitrust-alliance-...

Yep, and yet I’ve been able to successfully implement it in a 1 year project in a prior org (as part of a team obviously). HITRUST isn’t that bad, and it’s better than the alternative, which is HIPAA directly. I would best describe HIPAA as Vague, Fruitless, Bureaucratic, and Arbitrary. HITRUST is a huge improvement even if it’s not perfect.

Wasnt HIPAA not intended for security or privacy when it was originally developed? Merely as a standardized approach so various vendors could integrate easier. I could be mis-remembering this though.

Yes, that’s accurate. It is not prescriptive at all, but it does contain broad data security requirements. These are really the only legally mandated security requirements in healthcare. That said, HIPAA is more about establishing a legal and contractual framework for sharing data between providers and insurers and different providers.

Cyber security standards are in place to make the process easier to understand for the non-technical executives, who approve the budgets.

Without the standards the executives don’t know who they should believe, and invariably they believe the guy who sounds and acts like themselves, which means he knows as much about cyber security as the executives.

If you know what you are doing regarding cyber security, AND you are doing all the right things, HITRUST compliance is a cinch.

If you don’t know what you are doing regarding cyber security, HITRUST at least gives you a fighting chance. But then that’s the rub, if you don’t know what you are doing why are you running cyber security.

And how does this seemingly absurd exercise make hospital information systems more secure? These non-technical executives are also probably not aware of the intricacies of a knee replacements surgery... and shouldn’t be... these execs should be hiring and trusting skilled practitioners both in the operating room and in the information security dept. NOT injecting their ignorance of these disciplines into the process of ensuring good patient outcomes or security of patient information.

I think they are intended to be helpful, but they are adopted as CYA that have the side benefit of improving security.

> that have the side benefit of improving security

Sometimes. Other times they have the side effect of worsening security, as line employees have to deal with bullshit "security" rules and invent undocumented, untracked workarounds just to be able to do their jobs at all.

I work in health tech (full stack insurance), and sit next to security and IT, so this is a frequent topic of conversation for us. :)

For some context, this is one of our favorite websites/datasets: https://ocrportal.hhs.gov/ocr/breach/breach_report.jsf.

It is a structured archive of all reported health data breaches, major or minor, over the last 15 years or so, as reported by the breached entities. They’re required to report breaches as part of HIPPA compliance, or something related to it.

It’s a fascinating quilt of stories, with patches for phishing, accidental email attachments forwarded, and rogue admins. Fun reading. You can also load it into sqlite and find some interesting results (leakiest companies, states with most breaches reported, etc).

Hospitals might be a weak spot, but at least their weaknesses are ruthlessly well documented! As opposed to, say, financial infrastructure which IME is a similar horror show of monkey patched sftp servers.

Solving this collective technical debt is a massive coordination problem. It’ll be interesting to see if we ever get there. My suspicion is that the changes will be driven by monopolistic insurers, if ever, since that’s where all the money comes from (if you go to doctor at hospital X, your coinsurance will be Y instead of Z, because doing business with X is more/less risky due to their documented data practices). But it’s just a suspicion, this kind of thing might not be solved in our lifetimes.

The central IT function in a US hospital also usually has little organizational power and funding. Admissions, radiology, etc, buy whatever hardware and software they want, and the underfunded IT department has to figure it out.

Not to mention that network security necessarily means limiting access, and getting that wrong in a hospital context can lead to wasted minutes and hours that can cause harm to somebody.

This may vary by hospital, but in general many hospital IT staff tend not to be very good with computers, from my experience. Many are more focused on business/bureaucracy, or maybe they're just unskilled. I don't mean to attack their character, but instead to make the point that some very unqualified people are in charge of very important systems.

(Edit: My first job was hospital IT for a few months, and my boss was actually a pretty skilled programmer with a good grasp on security. So there are definitely exceptions.)

I imagine not many hospitals hire security talent either, or that they do much security beyond the "change your password" email every 6 months. Oh, and doctors/nurses/etc tend to ignore those emails.

Agreed with this. IT in hospitals is perpetually underfunded and basically a playground for creatures of corporate politics. Between administrative staff who think their medical credentials qualify them to micromanage IT decisions and perpetually under-funded departments I'm actually shocked that their systems aren't regularly crippled or destroyed by malicious entities.

Don't assume your medical data is secure. Systems that conform to HIPAA regulations are just one part of their computing infrastructure, and it's trivial to maliciously access a huge surface area outside of those specific pieces of hardware and software--and once a malicious actor has that access, it's not too hard to cross the gap.

Radiology has gotten somewhat better. They do might buy whatever they want, but the files tend to end up in a vendor neutral archive with proper access controls.

+1 to this and some other comments below. Our combination call/field technicians are actually quite experienced for help desk (min. 4-5 years experience) but have been perpetually understaffed. We have to go through a staffing firm to hire people at low rate to fill those ranks and then people wonder why they don't do well.

Upper level IT management doesn't communicate with the team when large changes are made, and pretty much treat them disrespectfully, even though they have their fingers on the pulse of what's going wrong from the medical staff. There's also a lot of waste from poorly implemented/delayed projects (there are more PMs than IT staff to implement the projects). Definitely a frustrating and bureaucratic area to be in.

Recently saw an ad for an IT support position at a hospital. The list of potential hazards in the work environment listed in the ad likely scares off many who have plenty of other employment opportunities. And most hospitals can't jack up the pay to compensate so attracting good talent is going to be a problem.

What were the hazards that you saw? Just curious

It's been over a year but I seem to recall potential exposure to radiation, infectious disease, and chemicals being on the list. It was quite long. I'm guessing the legal department added it for all positions in the hospital.

Yea that's just lawyer cya speak.

> And most hospitals can't jack up the pay to compensate

I find that hard to believe in an age of $100 saline bags, $20,000 childbirths, and 15-minute-long $500 specialist visits.

You’re downvoted likely because the high expenses of hospitalization speak more to inefficiencies in healthcare supply chain and the economics of insurance and really has nothing much to do with how much hospitals can afford on IT. Some hospital systems are rich... but they’re not that rich.

That said I agree (based on 1st hand experience) that the larger healthcare multibillion dollar systems in the US can afford to pay more for better IT/engineering. There is simply little incentive to do so. And further it’s more than just hiring a few engineers with FAANG pay... these institutions are organizationally not suited to engineering. Changing this would not be easy for them...and no, we don’t need a hospital run like Facebook or Uber.

Then there are the tons of smaller systems in the US.. they cannot afford high priced engineers regardless of the pre-insurance line charge for a bag of saline.

Much of that cost has to go to people who cannot pay for healthcare and the massive amounts of bureaucracy managing multiple contradictory medical billing codes by company.

Earlier in my career I interviewed for a health IT job that was basically a director level position. The pay ended up being less than I was making as a government employee for a smaller scoped job. The government gig was probably less than an intern makes at a FAANG.

In medicine, doctors are king. Everyone else is a peon.

Doctors don't feel like they are kings--while they make very good money there are massive amounts of red tape, filling out Epic...

It's the bureaucrats who are kings.

My doctor friends confirm. They would rather spend their time on patient care but have to make a lot more time for the paperwork.

"Sky is blue, news at 11:00"...

Of course hospitals are a security weak spot: They're full of sensitive patient health data shared over computer systems whose users and procurers are not very security-literate, and often absent-minded about such issues due to the grinding, stressful work.

waiting rooms are a gaping hole. nobody seems to see a problem with blabbing out your final 4 and first,last name when thier at a desk in a room full of whoever walked in and sat down.

un protected desktops are another issue, there is a tide of duties and an attacker can pattern the staff and get a good idea when they will have time to do an inside job of some sort.

As with most environments, there’s a lot of trust based in a hospital running successfully.

At least they have their own on-site security that’s experienced in taking people down.

I continue to believe the real threats are actual insiders and remote attacks.

Dunno how far someone will get with a USB key versus sending everyone a plausible email.

>Dunno how far someone will get with a USB key versus sending everyone a plausible email.

Insiders still can be threats. There was a machine that was deployed in a hospital for clinical imaging that some rad tech who guessed the administrator password put folding@home on without telling anyone which crippled that machine's ability to perform its function.

> some rad tech who guessed the administrator password put folding@home on without telling anyone which crippled that machine's ability to perform its function.

How incredibly bizarre to do something that dumb for no personal benefit.

F@H had value!!!

I do remember an IT admin day that said he ran SETI@Home at a low priority on all machines because detect any problems with a machine (e.g. spyware, crashing, heat problems, etc.)

But 2002 thinking wouldn’t fly in 2019.

BTC miners occur more than F@H these days, but they happen plenty.

This incident occurred before crypto mining was a thing many knew about. He thought that since the machine was unused overnight that someone should get some benefit from it.

These aren't mutually exclusive vectors of attack, they all need to be addressed.

You plug in the USB key, then you pull out the USB key.

The physical security layer at alot of hospitals is almost entirely absent, sadly.

What I meant was that sending everyone an email will get you further with less time/effort than actually going.

or you swap keyboards with a special keyboard [maybe a pineapple?] , or you can swap ethernet patches around.

Given how terrible a lot of low-end Dell keyboards get after years and years, most people would cheer :)

With the main apps being virtualized, workstations are refreshed less often than they used to be.

USB keys are blocked mostly these days. There are other huge vulnerabilities if you have physical access and are motivated.

From experience in plenty of industries, your statement is incorrect. Most places suck at security and blocking removable storage, but likewise suck at far more important controls (eg application whitelisting) for it to really mitigate much in the first place

In my org's environment, not very far with USB key. Email = very much yes.

We had one user who called after filling in every email address she had into a very plausible looking O365 login page. She admitted she initially distrusted the email/link that led her to this page and had replied saying so. The hackers on the other end told her to go ahead and do so. I mean, who she to question when it's coming directly from the hospital's lawyer?

Specialist I went to attempted to collect a photograph in the waiting room, as well - "please hold still a second while I take your picture for the doctor", with a webcam sitting atop the counter between us.

these are times when i say no thankyou thats not medically necessary, and obsruct the cam or turn around if its unobstructable.

> attempted

The big threat used to be the paperwork (and still is sometimes due to fax machines). There were people who broke in after hours, and stole boxes of paperwork to use for medical billing fraud.

We're past time for simple challenge response for this, if not something better. Computer picks two digits, maybe they are part of your SSN sequence, or not, you have to parse that and say true or false. Then last three of your SSN. The current strategy is b.s.

Same thing for picking up prescription at CVS/Walgreens.

They make you verify your phone number and address. Every. Single. Time. In public.

It's a shame how silly it all is.

I actually feel kinda sad for you, seems so cumbersome, yet I can't relate at all. Really bizarre to me as an Estonian, I've only had to show my ID-card to the pharmacist and get my prescription, because my doctor has entered it into the e-prescription[1] system. Reading this thread definitely made me appreciate it a lot more.

[1]: https://www.eesti.ee/eng/services/citizen/tervis_ja_tervisek...

It seems the biggest reason they're a weak spot is that the data they store make them a target. Retailers are also weak on security -- really, I wouldn't trust any company that wasn't a specialist in the space, i.e. finance and tech -- but most entities don't know so much about their clientele. Retailers don't need to keep as much info as they do (aside from profit motives), but hospitals probably do, so I can see this being a vulnerability that's never closed.

Honestly I have to completely disagree. Security is simply bad.

Hospital IT depts are pulled between many competing interests which lead up to this.

The data they have are sensitive, but that's just the reason they're a target. They're a weak spot because of poor security practices, which is due to poorly managed IT organizations, which is due largely to the egos of administrative management and poor funding.

So are vet hospitals. At this very moment there's a chance you'll walk into one that has fallen back to paper records and billing due to a continent wide ransom ware attack.

https://www.reddit.com/r/msp/comments/dnd7aq/ransomware_atta...

From that thread: Avimark is an old style load the EXE from a share program with a flat file structure for the data. Most clinics are not in a domain, just workgroup, and the share is read/write access for Everyone. So, yeah.

It's worse than that thread reveals. NVA was hit by a ransomware attack in May. They're now in a second attack that began in late October (ongoing). The latest one was described by CIO Joe Leggio as a "coordinated and sophisticated" attack in an internal email. He said it was designed to breach the NVA system specifically and that the attackers had three separate entry points. Only this week did NVA deploy endpoint security software to every computer in their 500+ veterinary practices.

Note: Avimark itself is not at fault here. The Avimark issue that the practices are having is related to NVA not having a solid DR plan with working backups. Part of the problem there is that because of Avimark's architecture, most practices have an on-prem server that each workstation RDPs into for using Avimark. Because this equates to 500 or so Avimark SQL Server instances spread around the United States, it's perhaps not surprising that NVA's unsophisticated IT department did not have working backups for each instance.

This sounds exactly correct unfortunately. Also taking down clinics in Canada.

This sounds ideal for a SaaS. Why is each practice messing around with an IT dept and SQL and DR when it could be hosted and managed at low cost for all them at once?

The industry has been really slow to move to SaaS. Avimark's primary competitor has a strong SaaS offering with Idexx Neo [0], but NVA requires the practices they buy to switch to Avimark.

[0] https://www.idexx.com/en/veterinary/software-services/neo/

Hello. I'm a reporter at the Wall Street Journal and I've been looking into this second attack for two weeks now. Some users from that Reddit thread have passed along some internal emails to me, and some customers have reached out with complaints as well and I've been looking to corroborate some details. Would you be willing to chat with me via Signal?

AVImark does not use SQL in general. They did an experiment with SQL a few years ago but abandoned it. They have a flat file database written in Delphi.

AVImark is not SQL. AVImark runs at workstation as a UNC path shortcut and no application is installed on the workstations. Entire program resides on the Server and is much like it's DOS predesessor from the 1980's. Everything it needs to run is installed in it's AVIark folder on the Server. Very few of the 8,000 plus hospitals run it in an RDP mode. Usually those that do have multiple locations with satellite hospitals connected to the main server via Internet back to the central or main hospital. Problem is NVA does not know what they are doing for security and has paid no attention to this problem, and has no well defined distaster recovery plan. They do not allow for outside expert AVImark or IT consultants to help their hospitals.

Dr. Paul DVM and AVImark Consultant and Trainer since 1998.

I feel like Mr. Robot may have highlighted this fact (along with others) to the general population rather effectively.

https://www.youtube.com/watch?v=g6gG-6Co_v4

Given the state of cybersecurity right now, is there any organization or domain AT ALL which is strong and model-worthy when it comes to cybersecurity?

Big tech. Google, especially.

How so?

In the wake of discovery of attacks by China[0] and the NSA[1] Google has adopted a nation-state actor threat model and a siege mentality around data access and encryption.

[0] https://en.wikipedia.org/wiki/Operation_Aurora

[1] https://www.washingtonpost.com/world/national-security/nsa-i...

Any idea if that siege mentality is part of the reason for sunsetting its social projects?

To be honest, Google is the last company I want handling my health data. If you don't check the right boxes, it could end up being "anonymized", and sold off.

Privacy is not security

Google is very good at precisely controlling what happens to the data. You never hear about some huge leak where 1B google accounts had their whole data taken.

There have been two reported Google breaches, both small.

https://www.oag.ca.gov/privacy/databreach/list?field_sb24_or...

Curiously, the "data breach" for which Google+ was supposedly shut down ... is not listed.

Everything in US is targetable. The main problem is that say the power/health/<fundamental infrastructure> are all managed by 1000 different companies who are all at different wavelength as far as OPSEC.

For those interested, I wrote a primer on M aka MUMPS at https://learnxinyminutes.com/docs/m/

Possibly in one part because I see people on freelancer marketplaces making software for hospitals, with job budgets of a couple hundred bucks. I'm ok with freelancers in general, but I feel that integrating code from disparate small jobs while keeping security in mind isn't gonna be so simple.

I’m an IT guy; I cringe almost every time I interact with the healthcare system.

I could pile on; all I want for now is encrypted and signed email with my doctors. I have an S/MIME certificate; can’t see why the IT staff at the hospitals I deal with can’t make sure my doctors have the same.

Because doctors are spoiled children. Were rolling out keyfobs for 2FA for our e-prescribe solution, but I'm keeping the fobs because I KNOW the docs will forget them/lose them. Docs only get soft-tokens on their phones because they never forget their phones.

A different hospital/security thread from a couple days ago: https://news.ycombinator.com/item?id=21483337

How do they store their data? Why don't they use private clouds?

-Duple? https://www.duple.io/en/

-Nextcloud? https://nextcloud.com/

IBM,Cerner,Dell

Reply on news.ycombinator.com