Regarding first paragraph of the story (redundant pair or routers crashing at once), it worth mentioning one possible neat trick to avoid this: When having a redundant pair of devices (routers, etc.), make sure such pair is comprised of devices made by different vendors - thinking here being that if there is a bug that triggers router crash, it's unlikely that two different vendors would be vulnerable to the same one.

Or with a generous budget, you can do what BT have done with their 21CN project: For each macro-block (access, aggregation, core), have two redundant networks each one built by a different set of vendors.