Baltimore Blew Off Ransomware Demand Only to Find Data Had Never Been Backed Up

First of all, it's a terrible article, it's full of typos; why aren't we linking to the source article on ArsTecnica?

Secondly, Later on in the article is the more relevant

>The city figures it will cost $18 million to recover from a rejected $76,000 ransom demand. I guess if you're going to play chicken with extortionists, you might want to make sure your backup plans at least meet min spec.

Either way, it's a terrible article and I'm on the fence about flagging it.

I wonder if, by the time we factor in all the costs of putting everything on a computer and online, securely, it's going to look like it would have been a pretty alright idea to leave most of it the way it had been in the distant past of 30ish years ago.

Just my personal opinion, but I figure the logical response would be to audit 1) the state of their backups, and 2) the source of the malware. After confirming that all necessary data had indeed been backed up and could be restored, and after blocking the vector used to deliver the malware, they could refuse to pay the ransom and restore from backups. Then they could go about further hardening.

If it was discovered that the backups were faulty (as in this case), or if it were not possible to discover how the infection happened prior to the ransom deadline, I would have advised paying the ransom, since 76k is a relatively small amount to the city of Baltimore. (And then of course continuing to investigate and harden to avoid having to do so again.)

People have principles when it isn't their money. The best route would be to pay it, then divert the money and fix the issues. Think of the 76k as cheap penetration testing.

>I'm curious if that remains true?

It's complicated, like any scenario where there is individual benefit but social harm (with healthy mixes of fault and incompetence too). It's so obvious sometimes it can get lost, but the people behind the ransomware are criminals. Ransom paid to them is money that is in fact directly going to supporting criminals, the government and in turn taxpayers aiding them. At "best" they're very pure single type cybercriminals and it "merely" supports them in pushing more ransomware. More often criminals are into a diverse set of activities, ranging from other varieties of cybercrime to being part of a real world physical syndicate, in which case some of the ransom is going to an organization that does stuff like murder/assault/human trafficking/drugs/etc. The strong reactions against just paying the ransom, even if it's cheaper and a clear winner from the individual first order perspective of Baltimore City, stem from this and that hasn't changed. Even if it costs $18 million vs $76k, that is still $0 directly going to the criminals themselves. On some real level, if Baltimore just paid the ransom they would be attacking the rest of us for their own benefit following their own fuckup.

Sometimes there might be other mitigating factors in the intensity of people's reactions, but most of them don't apply here. This wasn't an innocent or complex mistake, it was utter, mind blowing incompetence at the most basic level. It wasn't a matter of economics either, or technological sophistication. They'd be immensely better off if they'd merely aimed at any basic network share and cycled a few USB hard drives into a safe deposit box twice a month. That'd still be embarrassingly primitive and ineffective by any remotely modern standard for any significant organizational entity, yet it'd be something. And since it's a democratically elected government entity the complaints about it falling on citizens fall a bit flat too, because citizens have a duty to be checking on their government in a democracy. And this has been a lesson that has helped push things forward elsewhere at long last, so some good came from it that wouldn't have if the ransom had been paid.

So basically yes, it remains true for a lot of us.

I work with a cryto exchange, and fwiw, can tell you that most businesses would agree with you. They pay the ransom. And they use common exchanges like Coinbase, Binance, and Bittrex to do so.

The only way not-paying the ransom is beneficial (aside from having proper backup systems in place obviously) is if there was a large public sentiment shift promoting NOT paying ransoms. If society at large came together and decided the majority of entities are NOT going to pay, then ransomware viruses would be less profitable and thus not as favorable projects for hackers/scammers.

But that would take a lot of effort, organization, and favorable circumstances to have everyone do that simultaneously going forward. And there would be casualties at the beginning before the public sentiment cemented itself in the collective conscious.

But yeah, the only way ransomware will stop is if it stops becoming profitable: which means companies either need to have proper OpSec and backups (so they have no need of paying), or collectively agree that no one will pay ransomware attacks.

Seems like a pipe dream that we'd ever get to that point though. So I imagine companies will continue to fork over the ransoms.

Edit: this got me thinking, say for example, the US government outlawed paying the ransom to hackers. And they could somehow enforce this law effectively. Wouldn't that pretty much stop ransomware attacks in the US? or where ever a law like that could be effectively enforced?

Paying ransom makes it profitable. If you make something profitable there will be more of it as there is an incentive for individuals to continue those attacks.

Honestly lack of back ups is unacceptable. I can accept it if its a young start up handling some basic data and passing payments to an established business. However, any SE, DBA, IT director/admin with some experience should know about backups and push management for them. If management fails to pay for cost of backups, that individual should note it(preferably with a paper trail) and bring it up when shit hits the fan.

I will write SQL statements prone to injection, I will hack things together when I have to due to circumstances but I inform management of risks and they make the decision to proceed in a certain way knowing the risk. If they try to light my ass on fire 2 years from now, I will open the email and say "told you so". I have actually done that multiple times already and never got burned.

> paying the ransom is no _guarantee_ you actually recover

I’ve only helped recover from ransomware once about 2 years ago. When we ran the decryption program provided after paying it decrypted ~97% of the data. Some files were just permanently corrupted. Windows Server with ECC memory fwiw.

Fun note but the owner wanted to reboot after decryption and I yelled “No!” across the room, wanting to clone the known good data first. Good thing too, a reboot started the encryption all over again… good times.

I heard they were hit again but took my advice and setup Backblaze.

AFAIK everyone who has paid the ransom gets their data back. It's a pretty slick operation with decent support after the ransom is paid.

Don't negotiate with terrorists.

It depends... if the decision makers knew that there weren't solid, tested backups, they should have paid, cut off all external access. and worked to lock things down.

Also, regarding backups... at least 3 physical devices and at least two distinct geographic locations. Have backup systems push to a drop location (temporary), where a pull system can then pull those backups to a separate location. This provides a separation between your internal systems and backup systems. Otherwise, in situations like this, the backups could have been compromised as well.

Aside: this is a perfect example of why government agencies should have policies in place for disclosure of security breaches after 60 days or so. This allows them a window to exploit as needed, while still preventing catastrophe in the case of too many previously unknown exploits going into the wild in a breach. While I'd prefer everything just responsibly disclosed to affected software, I understand that cyber warfare in state actors is a thing.

In addition to what other comments said about it making these attacks profitable, 76k is not nearly a big enough stupid tax for Baltimore.

We do this. Every place I have worked at I've had the senior admin explain and show the backup plans for servers and desktops and demonstrate various restore requests. I have caught a few with their pants down.

Another important thing is fine granular permissions on network shares.

Penetration tests are a huge deal right now. Many companies outsource it to security agencies who then perform the pen tests and report results.

I briefly worked for a school district that had 8000 Macs to support. They fired the entire technical team (for reasons unknown) except for me and then proceeded to hire people who had never used a Mac in their life. The woman in charge of the department was like Lorraine Bracco’s character in Hackers - extremely overpaid, could barely figure out how to work a computer, and used God as her password. I’m still disgusted when I think about how corrupt and incompetent that district was and likely still is. A few of the fired guys wound up with a nice settlement, I’m sure the webcam recordings of the superintendent exchanging teaching positions for “favors” probably worked to their advantage.

I would take that a step further and say, "If anything, including automation, can tamper with your backups, they are not backups"

Several companies have gone out of business this year due to getting their servers wiped by an attacker. Their backups were on live servers as well.

I used to manage a set of tape backup robots (huge square tapes, I want to say 25 glorious GB a piece, and some Sony DAT devices @50-75GB a piece) that backed up sister data centers over a few DS-3s back in the early 2000s. The hardware/servers involved were around a 3/4 million.

I only provide that context because I've been as far away from that experience as I can possibly put myself and I can only hope it's gotten a lot better.

Prior to me and during the time that I managed backups, the statement "if you don't have backups at all, you don't have backups" was true for the entire backup set probably about once a week[0]. Those miserable robotic devices would crush tapes, the drives would eat tapes or otherwise fail, the backup servers would be over-whelmed (despite being nearly the largest hosts we had), the software would crash (Backup Exec), the server would just be ... skipped? And the tapes had a lifetime that was 1/10th what was advertised (and 1/2 what we had planned on[1]).

Our process included a monthly restore test of a few critical pieces of infrastructure and a rotating list of other hosts. I think the one time that we passed might have been worthy of opening a bottle of wine. Most of the time we could get enough data back that the restore wouldn't result in more than a rounding-error financial loss, but for the most part our server team spent a lot of time praying.

That system gave me nightmares for years after I left that position.

[0] It was so bad that our initial plan was to backup DC A using DC B and vice versa, but we opted to backup both from both data-centers (one of our requirements was off-site backups).

[1] I think I read the headline to an article a month ago about why "tape is still king" in backups. I couldn't bring myself to click-in. I assume things have gotten better, but if they're still selling this product the way they did back then, I'd want real-world numbers on MTBF and the like.

This reminds me of the Gitlab outage when they realized their backup images existed but were size 0. If you aren't regularly restoring backups to a machine, you are setting up for failure.

Question:

OK, so I know how to test — manually — that a few randomly chosen files are correctly backed up in my backup systems.

But what if there are larger classes of systemic error in the process I haven't thought of testing? What if some particular file type or directory tree has vanished from the backups, but since it isn't in my manual testing process, I never catch it? Are there best practices for validating that whole directory trees are correctly backed up across the board? Or any form of automated testing for backups (but, presumably, separate from the software that does the backup process)?

Every night we rebuild our dev and qa servers and databases from backups of production which were taken that same evening. If there is something wrong with the backups, the dev team will definitely know about it the next day when they try to start working and find that their development and qa systems are not working.

Reminds me of a client's system I was on a few years ago.

They had a backup script on their server that was written some time in the 90's by the looks of it. Used ye old `cpio` command to write to a tape drive; someone's job was to manually go and eject the tape every morning and replace it.

For verification, the script simply checked the exit status of the cpio command, and if it returned 0, it wrote "BACKUP STATUS: SUCCESSFUL"-- or 'BACKUP STATUS: BAD" if non-zero--to a log and sent an email.

At some point we had to try to restore something, and spent several hours trying to figure out why their backup tape was empty. Some quick digging into the script discovered that the command would indeed write some kind of tarball to the backup tape, it was just zero bytes.

IIRC, I eventually traced it to some problem with the cpio command erroring out if it tried to copy a file larger than 4GB. Suffice to say, the client was a little shaken that they had gone god-knows-how-many-months without an actual backup.

We are best off not paying them regardless: if there is no payout the bad actors will give up the attack. If you are in a group that pays out target them. It doesn't pay to target someone who has good backups (though it might not be worth checking if infecting everyone is easy)

Some of that is going on. I've heard of some orgs being asked to pay a ransom in hopes of picking up a trail to the perpetrators following the BitCoin.

https://www.fbi.gov/news/stories/ransomware-abettor-sentence...

One problem with that... international borders.

I wonder how the IT director was hired and who monitored what the department was doing. I worked at a place that hired a MS system support person but no on one knew anything about it and he seemed to know more than they did. He was fired a couple months later as he actually knew nothing and spent all his time making support calls trying to get someone to tell him what to do. He was on the phone for hours every day. Then when he was fired no one changed passwords so he logged into the DNS server and randomized all the IPs, leaving the whole company without networking. Hopefully not the same person...

I would take the $18 million "cost" with a grain of salt. It's not easy to come up with an estimate for the actual cost of lost files and delayed revenue.

It could have been some trigger-happy accountant just tallying up stuff left and right.

Some organizations need a disaster to provoke change. Hopefully, they'll do the right thing now and transition to something that works.

Surprised someone would use or recommend FileZilla after their numerous adware issues.

First you'd have to solve politics. Good luck with that.

You can't do anything but let it publicly fail. These incompetencies are usually the result of the top not believing that the cost is worth it. Also a well run IT department often gets "nothing's ever broken, what do they actually do?" so you get rewarded for doing break/fix.

Apply for a job. Run towards the fire. Help fix it.

I worked in local govt as a consultant, then as their IT Director for a few years. Nowhere near as big of a place as Baltimore though.

The hardest thing for me to do was convince anyone to get me money. They STILL have servers running that were purchased in 2007-2008. (I moved all critical services off, and made damn sure backups worked). It took two years before I was able to convince the board to give me $5000 for server hardware alone. That went towards the most needed hardware too, not my wish list.

This was before I was able to get things set up so I was able to buy something under $500 without board approval, just needed IT Committee approval. The board required everyone in the county to go through the IT Committee for any purchase related to IT. Printer ink, a $10 wireless mouse, etc. That was the first thing I got rid of while I was there. They only reason they did that was because it gave them power over people. I've seen them turn down people over $15 purchases...

The other person who replied is pretty correct about politics. The board there HATED tech for no reason other than if you weren't in the "old boys club" you were garbage to them and they treated you like that.

So overall, it really gave me a great view on why so many counties are so dysfunctional and why government works so slow (for most things at least). I don't regret my time there, but I wish it didn't take such a toll on me.

/Edit Also, there's no reason that they couldn't have tested backups. I was the only IT person for a county of 350+ PCs and a hospital, and I still managed to test backups every month. Shadow Copies saved me so much time too. I also used linux /pfSense when applicable.

I am not sure it's just IT competence. If this was the worst thing going on in Baltimore it would be doing much better as a city, but its not.

Consolidate with counties (for larger/populated counties) or the State. Bigger public sector entities are as effective as a comparable corporation.

Municipal government doesn't have a sustainable funding model for things like IT unless they are really small or really big. In my city (~100k people), they have 3 guys and a couple of freelancers that work <100 hours a year. The staff make like $40-55k/year.

Let me put my cynical hat on: legalese ransomware. Codify in law what is the fair price and a whole cottage industry will spring up to do the education for you.

Stop these positions from being political hires. The IT head had a political consultant, and was the highest paid person in the city government. He was hired by the former mayor who is now under FBI investigation.

Increase pay and reduce hiring friction. Most people with good skills aren't willing to do the long waiting game or take the pay cut to work in the public sector directly.

Never pay a ransom: it just encourages the bad guys to try harder. Soon they will figure out how to corrupt your backups as well (if there is anyway to do that - now we are in an arms race). Better to write off the loss now and ensure the bad guys don't continue to think of new attacks.

There is one exception: if the ransom is paid in such a way that the FBI (or equivalent) can track where the money goes and thus arrest the criminals.

True, but managers also usually fail to account for the time and cost of remediation after a pentest.

Had the city engaged in the pentest, would there have been an appetite to spend 10-20x that amount on remediation?

For personal ransomware, the requests are average around $250, lower than you'd think.