British Library cyber incident review [pdf]
bl.uk
bl.uk
> These legacy systems will in many cases need to be migrated to new versions, substantially modified, or even rebuilt from the ground up, either because they are unsupported and therefore cannot be repurchased or restored, or because they simply will not operate on modern servers or with modern security controls.
> There is a clear lesson in ensuring the attack vector is reduced as much as possible by keeping infrastructure and applications current, with increased levels of lifecycle investment in technology infrastructure and security.
> Our reliance on legacy infrastructure is the primary contributor to the length of time that the Library will require to recover from the attack.
A lot of lines like the following, also indicate to me IT was increasingly were involved in fighting fires and maintining operational systems ("keeping the lights on") rather than deploying new infrastructure and automation, updating software etc.
> Some of our older applications rely substantially on manual extract (...) which in a modern data management and reporting infrastructure would be encapsulated in secure, automated end-to end workflows.
Modern business is IT, I know that I am preaching to the chior but this sounds a lot like their IT was seen as a cost.
Some aspects sounded quite interesting, but these weren't places pushing the envelope in any aspect of technology. I'm sure they were running outdated software and configurations on everything, but IT was closing their tickets and meeting their SLAs. And with no disrespect, these people weren't necessarily disruptors looking to shake up and modernize the museums' infrastructure and take it into the future either, they just did their job to the best of their ability and went home at the end of the day.
To generalize I find that this usually holds true in a lot of non-tech industries, and IT is generally seen as a burdensome cost as opposed to enabler of business.
While I'm certain they are underfunded and overworked, this sounds like they had an internet accessible terminal server. I'd like to imagine IT screaming this is a bad idea but a suit somewhere saying they needed easy access for partners. I can only imagine how insecure the solution they replaced with this one was.
Things like finding out the status of a renewal involved finding a fax machine, everything but the most trivial renewal (say, renewing if you're on a work visa) seems to be done in person with handwritten paperwork, and the amount of busywork that seems to be done by hand by the DMV agent is quite easy to blame for the impressive wait times, multiple hours even if you have an appointment.
My DVLA renewal was trivial comparatively, they could even use my passport for an updated ID photo. But maybe if you're not a UK citizen they also make you jump through weird hoops?
I'm not saying that the DVLA is good, just that it could be even worse.
Some things they do reasonably well, yes. But edge cases like mine are the pits. To get a licence back after a medical suspension involves DVLA and the NHS posting physical letters to each other! It took 5 months for this purely admin process to complete, after I was medically fit. Grrrr. That is a long time to be denied the right to drive.
> The Library utilises numerous trusted partners for software development, IT maintenance, and other forms of consultancy
> increasing complexity of managing their access was flagged as a risk.
> first detected unauthorised access to our network was identified at the Terminal Services server. This terminal server had been installed in February 2020 to facilitate efficient access for trusted external partners
Sadly their response seems to be using more cloud infrastructure and outsourcing more.
trusted != trustworthy
The essential lesson - that good IT and security people within your company cost money. It is worth paying for vigilance, loyalty and care - has not been heeded.
CYA - it stops being their management's fault if its outsourced,
What’s unfortunate is that they flagged this vulnerability in 2022 and planned to review it in 2024 ???
Does it usually take this long to identify impact of users? They mentioned they paid for identity protection for their staff & ex-staff as well.
Credit monitoring is usually offered as standard when a breach occurs, the UK is much less litigation friendly than the US so in the absence of any actual harm, that would discharge most of their obligations to protect you following an incident.
Price of everything and value of nothing. Outsource everything, underfund everything from systems renewal to staff salaries.
Take a look at the US DoD, NASA, etc. They love acronyms, complicated internal organisation structures, just as much as the Brits do.
So refreshing.
Ouch.
I see a few comments indicating that connecting Microsoft (? not mentioned anywhere in the report??) t Terminal Services to the internet was a wholly bad idea.
Aside: is the report using "Terminal Services" generically, or do they mean that the server hasn't been updated since before 2009 (? when it seems Terminal Services became Remote Desktop Services (RDS))?
Is there something inherently insecure about remote desktops, or is MS software here known to be particularly insecure, or ...? RDP is default enabled on MS Windows installs (I always disable it), is that more of a problem than one might imagine?
Do they say anywhere where the access was from (maybe only GCHQ know that). Presumably the firewall would only allow known connections - did they report on analysis of all the remote clients?
Exposing RDP to the Internet directly has been frowned-upon because of the attack surface being presented, there's no two factor "story" out-of-the-box, and you're opened up to brute force attempts on cruddy user passwords.
Older versions of the Microsoft Remote Desktop Protocol had a much larger attack surface than current versions. The current versions with Network Level Authentication (starting in Windows Vista/Server 2008) present a smaller attacks surface. Older versions used "homegrown" Microsoft crypto, whereas current versions use TLS.
Disclosure: I made a FLOSS fail2ban-like tool for RDP many years ago[0]. I had a situation where I was forced to expose RDP to the Internet and I didn't like having it open w/o some protection against brute force attacks. This tool happens to still work in Server 2022 and will slow the velocity of brute force attacks. I still highly recommend not exposing RDP directly to the Internet anyway.
(The ts_block tool is missing some fairly essential functionality that I never got around to implementing. It works fine and is really easy to install but some things are sub-optimal.)
There is a huge difference between a port forward on port 3389, and publishing the gateway behind azure app proxy - the latter supporting mfa, account lockouts, and not actually requiring any open port to the internet. Much of the discussion online treats these as equal.
¯\_(ツ)_/¯
For a report from British--and a library, no less--the lack of Oxford comma cocnerns me.
> In common with other on-premise servers, this terminal server was protected by firewalls and virus software, but access was not subject to Multi-Factor Authentication (MFA).
"Jisc is the UK digital, data and technology agency focused on tertiary education, research and innovation."
State-owned quango asleep at the wheel. Unsurprising.
In this case, it looks like Jisc was basically turned into a charity in 2011, so technically they're not even state-owned anymore.
This used to be what we called JANET. Back in the day this was top banana and prestigious to work for like GCHQ etc.
I expect they've died from a thousand cuts under the Tories. Every university I've been in the past 10 years have their ICT run by Microsoft, and is absolute rubbish.
My ISP could do the same thing. How is that being asleep at the wheel?
No root cause. On other forums it is understood they were running very old and unpatched VMware os. Which is simply embarrassing and everybody within their IT team should be fired immediately for gross negligence.
They can't inform people whos data has been compromised because they refuse to pay the ransom and have no other way to tell what was stolen. Farcical.
Their ability to rebuild in a timely manner was hampered by not having any spare servers and presumably because all their server hardware was compromised and couldnt be used for restore.
That doesn't fit their claims on page 7 about reviewing the lost data and contacting affected users.
At least they seem to have a plan moving forward that seems considered, though I think a lot of what they want to do is easier said than done effectively. I wish them the best of luck.
It said that. The terminal server entry point was completely scorched in the attack. Offsite rlogd would have helped.
It's bad that they don't know what was taken, but as for paying the ransom, I wouldn't do it either: first, because it's danegeld; second, because you're just exposing yourself to even further risk by accepting files from criminals; third, because as others said, it would be UK tax money.
As an organisation forming part of the UK State, they're not allowed to. Rightly, in my opinion.
Since I dont trust the library to actually assess my impact, or track records of companies getting hacked often drag their feet making it up to victims. (equifax)
The IT team most likely begged for years for funds to upgrade their infrastructure, but did not receive any of it. Public institutions are already short on money, but education has it even worse.
If anyone is to blame, it is the last British governments, who have focused their attention on Brexit and Ruanda crap instead of providing services for the citizens.
That may be true, but by that standard about 90% of every sysadmin, IT managers and even CISOs would be out of a job next week.
Most companies are just "getting by" and hoping it won't be them next.
We have a multi-national cybersecurity crisis due to decades of kicking the can down the road, excusing poor software engineering to allow unfettered commercial development, and destroying our education and training sectors.
But we've utterly normalised digital ignorance and built what Edward Snowden very rightly calls an "Insecurity Industry".
I'd go further, we've turned a celebration of ignorance around cybersecurity and dismissive attitudes into virtuous slogans.
"Don't make me think" - Krug
"Move fast and break things" - Mark Zuckerberg
"If you've nothing to hide you've nothing to fear" - J Random Idiot
What it boils down to is that presently there's more money and power in insecurity than there is in security. Our industry has multiple principal agent, Shirky Principle and Pournelle's Law problems, see [0].
We allow ransomware and stalkerware companies, and outfits like NSO (which I only mention because they are most well recognised) to operate as legitimate.
We flood markets with defective IoT crap and reduce consumers expectations to the level of accepting vendor malware and backdoors installed out of the box.
And then we turn around and complain that "stuff ain't secure".
This whole ship is DUI.
> "Don't make me think" - Krug
That quote has nothing to do with cybersecurity, it's the title of a book by Steve Krug about web usability.
I am unfortunately old enough to have read that book when it first came out, and it's exclusively around how to design front-end UIs on websites to reduce user complexity. There is no mention of infrastructure or security at all.
You're making a quote around how we should make websites more usable and understandable to users - so they can use them without thinking - into something it isn't.
It has everything to do with it.
I know exactly what the book is and I read it. It's actually an excellent book on UX and I expect Steve Krug picked the title because it sounds cool.
No disrespect to that author intended, but it (maybe unwittingly) expresses a sentiment that has grave implications about the position of technology in human affairs. To understand why, please look deeper into what we used to call Human Computer Interaction (HCI) or "Cognitive Ergonomics".
I think I recently mentioned it in this online chat [0]
Explicit cognition is the "thinking slow" part of our brains that uses so-called left-brain linear reasoning and logic. It sits high in the cognitive stack. But as people use devices today, in what McLuhan [4] or Innes [5] would call an "acoustic" (nothing much to do with actual sound) way, we drop down a cognitive level to a faster, visual-haptic loop that bypasses explicit reasoning.
Designing applications that bypass this has major effects on security. The work of B J Fogg will show you more about this [1].
Tristan Harris also has lots on it [2,3].
One of the disastrous effects of this "distracted" level of HCI is that people use more emotional cues, rote, colour, word association, implicit trust and other models that make them easy prey for phishing and other kinds magic and trickery.
If you're interested in a much broader understanding of cybersecurity I give you a sincere invitation to check us out here [6].
[0] https://www.youtube.com/watch?v=hYnOf4PWGpA
[1] https://behaviordesign.stanford.edu/people/bj-fogg
[2] https://www.youtube.com/watch?v=LUNErhONqCY
[3] https://www.wired.com/story/our-minds-have-been-hijacked-by-...
[4] https://en.wikipedia.org/wiki/Marshall_McLuhan
You do not really worry about what would happen if all the grossly negligent doctors get fired. Who will do those procedures with a total disregard for safety, said no one ever.
But I do. I care about them as people. People who have families and need a job. I'd rather help them to not be grossly negligent than see them fired (and probably worse idiots take their place since we are in a major skills crisis right now).
The world is getting complex faster than anyone can track. Tomorrow it could be you, or I who is getting called on gross negligence because we can't follow it. So I choose to be a teacher even though telling people the truth is getting REALLY F**ING HARD these days - cos no one wants to hear it.
You can train them once they are removed and reinstate them when they can do the job right, but supporting their continued harm of others so they can “support themselves” is detrimental, counterproductive, misguided, and extremely selfish.
You are literally better off paying them to do nothing. Please at least do that instead of paying for harm.
"should" is doing a lot of work there. Im so many ways we're in agreement. But I do this in the real world, and experience has shown me we must deal with the world as it is and not merely as we wish it to be.
They were following explicit government guidance, as promulgated by the National Cyber Security Centre (NCSC), which is the civvie offshoot of GCHQ.
> This report is a joke.
> No root cause. On other forums it is understood they were running very old and unpatched VMware os. Which is simply embarrassing and everybody within their IT team should be fired immediately for gross negligence.
> They can't inform people whos data has been compromised because they refuse to pay the ransom and have no other way to tell what was stolen. Farcical.
> Their ability to rebuild in a timely manner was hampered by not having any spare servers and presumably because all their server hardware was compromised and couldnt be used for restore.