Googler here - my opinions are my own, not representing the company
at the lowest level in case of severe outage we resort to IRC, Plain Old Telephone Service and, sometimes, stick-it notes taped to windows...
at the lowest level in case of severe outage we resort to IRC, Plain Old Telephone Service and, sometimes, stick-it notes taped to windows...
I remembered to publish my cell phone's real number on the on-call list rather than just my Google Voice number since if Hangouts is down, Google Voice might be too.
Backup tapes and in production servers are kept at different colocation sites to protect data from fire and other catastrophes of that level
Using colo sites on separate tectonic plates would protect you from catastrophes on a geological cataclysm level
Last time I used tape, we used Ironmountain to haul the tapes 60 miles away which was determined to be far enough for seismic safety, but that was over a decade ago.
What use is it if it runs on the same stack as what you might be trying to fix?
IRC does use DNS at least to get hostnames during connection. I'd be surprised if it didn't use it at other points.
My bet is, FB will reach out to others in FAMANG, and an interest group will form maintaining such an emergency infrastructure comm network. Basically a network for network engineers. Because media (and shareholders) will soon ask Microsoft and Google what their plans for such situations are. I'm very glad FB is not in the cloud business...
yeah if only Facebook's production engineering team had hired a team of full time IRCops for their emergency fallback network...