Hacker News Reader

Comment by mrkramer | Hacker News Reader

So somebody messed up Facebook's BGP records and traffic couldn't be routed to Facebook servers. I wouldn't be surprised if some angry insider(employee) got his revenge on Facebook for whatever reason.

View on HN

Could one single person really inflict that much damage without a balance/check or code deployment review process in place?

With the right access it wouldn't be hard for someone to configure some key routers in such a way that all traffic is blocked and no one can get into them over the network. They'd need to send someone physically to the sites to reset or replace them.

I'm still leaning heavily towards an "oopsie" with routing that accomplished the same thing, however.

Rollout gone bad is def more likely. People with that level of access aren’t stupid enough to get sued for tens of millions of $ in damages

It would not be the first time

I think if you were smart enough, you may be able to mask some needed changes under some legitimate tickets. You make certain changes that you know will break stuff, but you assign a reviewer who doesn't know enough about the particular thing that they may think it seems fine.

I am talking in a very generalized sense, not for this particular issue. But I don't think the code review/deployment process is entirely safe against internal bad actors.

Code reviews can potentially catch bugs and prevent an obvious inside attack but are mostly to keep the code-base healthy and consistent over time. Something that can take down multiple revenue streams for all customers should have some other check besides a peer code review.

See: The Underhanded C Contest.

http://underhanded-c.org/

The whole point is to write C that appears on the level at first, but actually has a subtle exploitable flaw. The flaw is supposed to appear like a simple mistake for plausible deniability. Some of the winning responses are very devious.

System administrator with high authority can pretty much bring down the whole network stack if s/he wants to.

In general, access levels are pretty liberal at FB (less so then in the past, but still liberal compared to others)

A network engineer with enough experience to handle Facebook's DNS and BGP configurations can probably design a plausibly deniable mistake/misunderstanding/unfortunate coincidence.

By the time it gets to that point, they may likely have already been identified as not a good cultural fit, unless they were a real sleeper.

I'm sure there are measures in place to prevent it, but at the end of the day someone almost certainly could and would have to be able to.

Reply on news.ycombinator.com