Wells Fargo outage includes debit cards

A friend of mine who works at WF told me they lost an entire datacenter to a fire. I can't find any 3rd party evidence to back that up, but I have no reason to not believe my friend.

I guess their catastrophic recovery plan didn't survive a real incident.

edit I guess fire suppression systems were activated in a datacenter and some series of events knocked the entire building offline and automated DR failed. So if that's true, at least the risk of data loss is probably less than if the entire thing burned down.

Perhaps it's time to force financial companies like this to "fail open".

Ie. If their computer systems are down and can't find a reason to deny a transaction within 5 seconds, then the transaction is allowed to succeed.

It can be reconciled later, and the consumer can still be held liable for spending beyond their limits.

Thats pretty intense. Their failover must have failed - I'm surprised that a single datacenter can take out debit card payments for a top5 bank. I feel sorry for the NOC team there.