Hacker News Reader

Comment by dec0dedab0de | Hacker News Reader

I haven't been following closely, but I think once they moved the prefixes they could no longer access the routers. Coupled with barebones staff at the data center due to the pandemic, and all internal communication being disrupted. Though I really expected it to be up within an hour or two.

View on HN

We have had out-of-band management ports & networks design for decades! I know the feeling of driving 8 hours because I lost connection to the device I was configuring. https://en.wikipedia.org/wiki/Out-of-band_management

HN engineers believe out of band admin control planes to be surveillance and backdoor firmware so they are disabled for privacy reasons.

Who the hell put HN engineers are in charge of facebook, and why haven't we gotten more out of it than a temporary outage??

Previously they had physical access to the data centers and weren't locked out.

This. And if you're really worried about it you can go crazy with security with individually issued hardware security tokens and one time use access tokens.

It's pretty inexcusable that FB wasn't able to use OOB management.

It’s likely that their oobm systems are depending on dns in some way - aaa for instance. It’s not possible that they don’t have oobm at all, really.

Yeah, I think that is true. If you look at the Update near the end of the Cloudflare article there is a huge spike in the BGP activity (I assume re-announcing all of the routes). So that part of it was relatively instantaneous after they got all of their ducks in a row actually getting to the routers and locating the BGP from some earlier version before it went offline this morning that they could use.

Reduced staff because of the pandemic? 18 months in? This looks like a sweet deal for those workers more than anything else.

Presumably they're working remotely, but the incident required physical access, so only the reduced number of people present were (immediately (or at all, I don't know their policies)) available to deal with it.

I read that on-site badge readers denied access to employees! (Not necessarily the critical team to resolving the outage, but still...)

Reply on news.ycombinator.com