Hacker News Reader

Comment by lol768 | Hacker News Reader

Existing discussion thread: https://news.ycombinator.com/item?id=14324129

---------

Shutting everything down seems like a really rash response, especially when these systems seem to be used for critical communication e.g. the phones too. The Twitter messages seem to suggest that doctors are seeing this on their personal machines, but why would this impact the phone system? Are they not separated out?

I'm also really curious as to how this started. The article mentions a "bug" in the IT systems - some sort of novel zero day in the software they're using that was exploited remotely? Or is it more likely someone screwed up and ran something without thinking?

There are reports on twitter that this is impacting X-rays, pagers as well as the phone system. This is ridiculous if true and suggests there have been some major failings when putting this infrastructure in place. Perhaps underinvestment in IT is to blame.

View on HN

> Perhaps underinvestment in IT is to blame.

Or, indeed, over-investment in trash-tier IT services provided by blood-sucking IT consulting companies.

I've seen the insides of some UK Government IT systems (not the NHS), and it's astonishing how little functional software one can get in exchange for a few hundred million sterling.

That, and the bitrot of holding on to ancient, never-updated IT systems.

[Edit], back on topic, I sincerely hope whoever did this is burned alive for their crimes.

The whole EHR market is absolute crap in Canada too. You have around 10 competing standards with no interoperability, multiple data sources cobbled together through webpage links, and all running on a slow as fuck central server that you Citrix into.

Another research department I worked at was seriously underfunded, which resulted in questionable decisions, such as using round cube for email, and a central shared drive with a Microsoft access database containing patient data. Hospitals have terrible security.

What is the issue or alternative to Roundcube? I thought they were pretty good.

It's a terrible UI for webmail, and our webmail was subject to a hack a few weeks ago, and they've since closed off WAN access. I'm pretty sure it was related to roundcube. Which is a bummer.

To be fair - healthcare IT infrastructure sucks everywhere, even in the country most well-known for remarkably expensive privatised healthcare. It's not really obvious at this point how to fix it. Have the Government Digital Service or similar work on it directly and ditch the contractors?

> Have the Government Digital Service or similar work on it directly and ditch the contractors?

Basically, yes. Bring it all in-house, ban the contractors/consultants/mercenaries/etc. Remove the profit motive and suddenly you don't have millions of dollars/pounds being siphoned off by vampiric consultancies and third-party vendors. Suddenly you can spend tax-payers money in a sane and rational way.

Hire a bunch of talented people who care about the wellbeing of their nation state, pay them well enough and task them with building the best systems possible in the most efficient way possible.

This is a great idea, but since it's practically to the left of Corbyn I can't see any chance of it being enacted.

Without giving too much away, I've been involved in just such an initiative, and it was _awesome_.

However, in the context of an established organisation it's really hard to pull off, and so we eventually ran into serious pushback from other factions within the org, particularly the established IT Ops folk.

Still, it can be done, and it can be a raging success. Especially gratifying when you spend two days writing up a system in Python which replaces some 90's garbage that's costing the organisation 200k per year in licenses.

Killing Leeches is fun.

Had exactly the same.

100 user system, Windows CALs and RDS licenses per user = a lot of money. Found only 40 users needed the CALs, rest were fine on Linux.

Took the devil of a job to persuade them this, as the Microsoft rep told them they couldn't.

In organisation like the NHS something like active directory becomes almost mandatory.

Open sources equivalents are nowhere near as good.

Not the same thing I'm on about. Can't say too much, but was on a system where for majority of users they didn't even touch a Windows server or AD.

Was that related to one of the NHS hack days?

> eventually ran into serious pushback from other factions within the org, particularly the established IT Ops folk

"Mordac, preventer of information services"

Thankyou very much for fixing this kind of thing, seriously.

This. A million times this. I think at this point the tech community should be vocal about this issue.

Tech has improved ten-fold this last decade, and IT consulting services simply don't care because they profit hugely from it. What makes this issue even harder to solve is the fact that IT is so simple to hide because society does not understand it enough.

Similar to the Tech Pledge, we should stand and be very vocal about the fact that you CAN build strong, secure and relatively cheap systems. If we don't do this, who else will ?

We're stuck between a part of the industry which benefit from this (and especially the big bosses, they don't care about the developers either) and a society which doesn't see the value of homegrown (as in company/government-grown) tech talent and the tenfolds decreases in IT spending it could entail.

Please, Sam Altman ? Someone ? Please ?

Can I ask, how did the employees fit in with the payment structure? My feeling has been that the NHS simply can't hire talented staff on the wages they expect because the pay grades don't go up that high. Hence hiring contractors.

It is my understanding that the DWP has brought a lot of its IT back in house.

edit: I should say some as to be fair, I'm not 100% sure of the extend of this.

My team at NHS Digital is building a replacement for the NHS website [1] in-house.

[1] http://www.nhs.uk/

This is what should have happened instead of the disastrous NPfIT. It took years and none of the big players (Accenture, BT, Fujitsu, CSC) could even agree what a fucking medical record should look like because nobody was willing/able to concede anything.

If it had been an in-house project with actual experts employed in building/deploying on a smallish scale (say, a town or county) and then rolling out it could have been a thing of beauty.

It should have been what AlphaGov became.

I agree with your solutiom, but there is far, far less blood sucking going on than you might imagine and far, far more organizational incompetence. That is, organizations hire contractors and then dont know how to tell them what they want, but simultaneously refuse to let the contractor have initiative because they dont know how to set up a pay structure for it. Businesses screw themselves.

Yeah but the truth isn't as exciting as thinking you can write DJango app to save the NHS. It's far more fun to think you can knock something up with your friends in a few months.

Then requirements start coming in, the stakeholders, the politics. The multi faceted organizations, the disparate teams with never ending edge cases.

3 years later when it's past phase 2 and creaking at the seams, along comes the next upstart... DJango! Which idiot picked that?!!! Me n my friends could....

> Hire a bunch of talented people who care about the wellbeing of their nation state, pay them well enough

This is impossible. Any large organization eventually resorts to using pay scales to combat corruption. When the right people will be 10+x more effective than the wrong people, pay scales are impossible.

Literally the entire reason why large organizations resort to hiring contractors is because they know it's impossible for them to hire good people directly.

How does it suck? Everywhere I look hospitals have dumb pc's connnected securely to cloud services for applications, primarily Citrix or web based portals hosted by a trusted contractor or the hospital themselves.

There's an awful lot of network-connected, really badly written, often unsupported software in healthcare, that uses proprietary formats. VDI doesn't save you if what's on the other end has to be Windows XP to run your shitty software that'd cost your IT department's entire budget for a year to replace and retrain users.

Or, indeed, over-investment in trash-tier IT services
Yeah, I think this is probably closer to what's happening.

I've worked with some of these Enterprise-level IT consultancies in the past. I do understand that it's quite a different market from the lean, tech-focused web development market, but some of the solutions I've seen implemented are shockingly* bad.

I wonder if it would be appropriate to "dramatically overreact" and send in the SAS or similar to send a polite message that mucking with the NHS is really not a good idea.

Maybe post offer a decent reward - say £10 million for information leading to the identification of the culprits.

If this trend of targeting large institutions continues, yeah, I can see someone eventually "overreacting" with black ops.

This said, they're probably popping champagne at Tory HQ right about now.

wouldn't they be popping corks at Labour HQ? I mean, this isn't really "strong and stable"...

Labour won't attack on this because they know it might have happened to them just as easily, whereas it fits the Tory narrative that "the NHS is useless and should be privatised" with almost no spin.

The NHS is fundamentally a socialist endeavour - any failings in it are obviously the fault of socialism!

Most of my local Tories know that any kind of direct assault on the NHS would result in instant annihilation at the next election.

There are more subtle methods of course including outrageously broken internal market management restructures (Stafford Hospital Trust, 'fund holding GPs' in the time of Thatcher) and the like.

I suspect that if this does happen it will more likely be the French retaliating against the hacking of the election the French version of SAS/DELTA don't fuck about.

Exactly this. When ever anything comes to tender, out of the usual three on the table, the cheapest will always be chosen.

And because departments are left to their own devices, they solve all their problems with shared drives and excel sheets.

Ransomware is the hero we didn't ask for.

Everything is a "bug" according to most journalists. We'll probably get something a bit more coherent from Ars Technica or The Register.

WRT the failings: I've worked in IT in the UK for more than 25 years, and I have never (until now) worked in a place that took security seriously. That includes schools, a large accountancy firm, several well-known public sector establishments, a political campaign, etc. "Optimistic security" is the model here, and hospitals have huge rambling networks with many legacy systems and third-party solutions. I would be surprised if they don't have security issues. Where they are secure, it's probably down to some unsung hero(es) somewhere, who took it on themselves to push security. (I've done this myself and it's a thankless task; nobody notices or cares. Dogs not barking, etc.)

> Everything is a "bug" according to most journalists. We'll probably get something a bit more coherent from Ars Technica or The Register.

That's a good point, tech journalism is usually pretty poor.

https://twitter.com/ShaunLintern/status/863039464649744384 suggests it's significantly more boring than a zero-day (though still incredibly problematic).

Your experiences with UK IT make for depressing reading.

> Your experiences with UK IT make for depressing reading.

I am astonished that even skilled techies don't take security seriously, using passwords like their car registration or company name - I've seen that with a military contractor, ffs. People who had signed the Official Secrets Act and had network links into supposedly secure sites. It depresses me too.

People have this insane notion that one line of defense is enough or they have the equally insane notion that no wants to take something they have.

People have a really hard time evaluating simple risk/reward models. You have to make the reward of attacking you higher than any possible reward to have a reasonable chance at security. If anyone anywhere on the Internet can profit by your loss, eventually someone will try.

EDIT - To make life even more difficult sometimes the attack provides gains indirectly. Imagine one group of politicians attacking a service supported by another political faction, just that service going down profits the first group if it changes who voters vote for.

From my perspective its a fundamental problem with incentives. Good security costs time and money, never gets return gains, and no one will put money up that wont ever see a ROI. So incredibly short sighted, depressing indeed.

Is there some kind of saying: You can't secure a network that contains legacy systems?

Start from scratch, don't connect anything legacy. Assume the LAN is already penetrated and design for that. Store no data locally, client machines run something like ChromeOS by default. Timeout anything that's not used for 6 months. Don't use passwords, only SmartCards. Snapshot data for ease of restore.

Don't you think that, perhaps, replacing a dependency on a centralized infrastructure by replacing the entire system by centralized infrastructure, owned by a foreign company, might not be the smartest idea ?

Systems that can't depend on the outside, obviously, cannot use cloud services. That means no chromeOS, no active directory, nothing but the local network. This is beyond obvious, and yet, I actually believe we'll be stupid enough to do exactly the centralization thing.

Paying $300 to these crooks, incidentally, will be a LOT cheaper than whatever microsoft or any other company will ask for the centralized infrastructure. Not that I suggest doing that, but still.

I did say something like, just take ChromiumOS and your run own servers.

All systems depend on the outside to some extent anyway.

Is it $300, or $300 per machine? Why can't the machines just be reimaged, what kind of giant corporation doesn't have that working automatically?

$300 per machine. Issue is that ransomware also encrypts all files on shared drives.

But a problem with the approach of centralization taken by something like ChromeOS which uses Google accounts (or I guess Win10S which uses ms accounts) is that you're attempting to prevent one player from holding you hostage by giving yourself hostage to another. This is not going to work to prevent paying through the nose, although yes, your new hostage takers will probably realize that the NHS will be able to pay more when it actually takes care of patients. Not too well, of course, good enough to make sure it isn't replaced or repealed. Badly enough so that constant complaints ensure a fresh budget injection every quarter.

They're already being held hostage by 1000s of companies who won't update their crappy software so it runs properly, allowing them to install OS updates rapidly.

As I said, like ChromeOS, it's perfectly possible to run your own servers, don't need to pay Google or Microsoft.

https://www.chromium.org/developers/how-tos/enterprise/runni...

> Start from scratch, don't connect anything legacy

The myth of the clean sweep. Personally, I've found such systems tend to be late, wrong and inevitably end up resembling what they replace, warts-and-all. This is because systems tend to mirror the organisational and political context they are in, and most programmers today are not significantly better than those who came before them. Quote me all the exceptions you like, this is my experience.

https://en.wikipedia.org/wiki/Conway's_law

Investment Banks take it seriously. So seriously some clown in IT Sec cut off the links to 3rd party libraries from the internal Nexus.

Ars story:

https://arstechnica.com/information-technology/2017/05/nhs-r...

We've merged threads with that original from https://news.ycombinator.com/item?id=14324303.

Voip phones run on windows servers! Thanks Cisco.

Reply on news.ycombinator.com