Comment by riskable | Hacker News Reader

Comment by riskable | Hacker News Reader

The article talks about all these sophisticated attacks that could hurt the financial system but I think they're not seeing the elephant in the room: What if someone just ran "rm -rf /" on thousands of servers at just one of the "too big to fail" banks?

How long would it take to recover from that? Would it even be recoverable?

I don't think any of the big banks employ enough IT people to recover tens of thousands of servers in time to save the world's economies from a major collapse. Recovering from such a situation would take months. Not only that but let's assume they hire temporary contractors to do the work... Can you even vet that many people that quickly? You'll be handing these folks some of the world's most sensitive information.

The Fed needs "IT stress testing" in addition to their balance sheet stress tests. They need to ask these sorts of questions: "If half your servers were deleted how long would it take to recover from backups?" "Demonstrate a backup recovery on the following ten randomly-selected servers right now. You have 8 hours. Good luck!"

Having worked at 2 of these "too big to fail" banks, disaster recovery testing is par for the course now. We would do DR testing every quarter on both the primary and secondary data centers. Common simulation scenarios were - network/disk failures in which case the secondary data center would take over the operations. We would also time how long it took for applications to come back up, with high priority apps (trading, payment etc) being given extreme importance.

We did this too. We tested our DR plan at least once every year (this included busing people to a different location, timed responses etc.), and we were considered a non essential part of the bank. We also had to make sure that all our servers had dual power supplies, and were attached to diesel generators in case of a power outage. They were much more on top of it than any other place I worked.

1. Offsite backups onto unpowered hardware are SOP for big banks.

2. Pretty much every transaction with the bank has a counterparty that is outside the bank.

It would be recoverable. But very very painful.

This is the "Mr. Robot" scenario, in case you haven't seen that show.

Hacking the HVAC system to raise the temperature of the tape storage facility to destroy magnetic tape backups in place was quite the idea.

Reply on news.ycombinator.com