Hacker News Reader

Comment by sync | Hacker News Reader

   On Monday at 3:30am PDT, we experienced
   a hardware failure in our primary encryption hardware device.
   The failure cascaded to the backup slave device as well.

How does a hardware failure cascade to a backup slave device?

View on HN

In the same way a slave database can be corrupted by the master when it has a failure. If you're encrypting data on the master and replicating to the slave, improperly encrypted data may be propagated before a fail-over. This is a case where having the master fail completely and go off-line is preferable to a "working failure".

This is why I use incremental backups, and check them regularly.

But doesn't a 'hardware failure' imply failure to a specific piece of equipment/hardware? A cascading issue with actual data isn't a hardware failure, it's a software failure.

One of the racks fell over onto the other racks, like dominios.

Ha - that made me laugh.

Ironically something not far off from that happened to a contract outfit I was working for. They had a telecoms rack rather than a proper one and they couldn't bolt 9x maxed out DL380G2 servers on it as it started to bend. They decided the best approach was to just pile them up on the rack bed.

This was until the one at the bottom needed a RAM upgrade. Cue 6 people carefully trying to lift up 8x DL380's in one go and sliding various OReilly books underneath. The inevitable happened - a hand gave way resulting in a pile of DL380s spread all over the machine room floor and a broken toe from someone in the hosting facility.

Nothing went down though!

"This failure corrupted encryption keys used to access stored credit cards to process recurring transactions"

Looks like they were sharing encryption keys by replication or shared storage.

"Flailover"

The primary fails because of a bug and so it fails over to the backup which flails-over because of the exact same bug, c.f. https://en.wikipedia.org/wiki/Ariane_5_Flight_501

Reply on news.ycombinator.com