Hacker News Reader

Comment by threatripper | Hacker News Reader

Do they share infrastructure?

Given the fact that it's also down, I'd say yes

It could be some rouge admin who actually deleted Facebook.

At that size of a company there's usually blast radius restrictions and per-role permissions. I don't expect anyone has enough rights to "delete Facebook" on their own.

I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue.

For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets erased because TEMPDIR isn't defined...

I guess they have enough pending stock to deter them...

This wouldn't "delete" Facebook or many much smaller companies. It would result in maybe a small outage and get restored immediately in most cases. It's also an infra change you'd need across many systems - this isn't possible as a single change "across entire fleet".

This is not how non-trivial services work.

That’s...now how any of this works. You can’t just change integrity-bearing things without FIM systems kicking in. And you’d need collusion to get something mainlined that would bypass that.

You don't need collusion - just a code reviewer not paying proper attention.

How though? Every past author of that script would be notified of such a change. It'd be insane if all of them would pretend they didn't see it and accept that change.

It would be hidden amongst a big refactor, and it would have a bunch of unit and integration tests that all work correctly (because they do set the environment variables correctly).

Writing code that appears to do one thing and actually does another is very doable - you only need to hide one malicious line amongst thousands in a code review, while the reviewer needs to inspect every line.

Besides, the vast majority of code reviews are 10 minutes or less.

I think you oversimplified the code merge process at Facebook. Their internal code review system (Phabricator) would automatically add tens of reviewers to your PR because you changed an important script authored by those people. Also, there are linters that would call out the use of potentially hazardous commands. Actually deploying the thing is another process that requires jumping through a series of other review hoops.

p.s. I had a short internship there.

Facebook servers do not have cron installed.

Example using widely understood tooling... But there are plenty of other one line changes with similar impact

one could hope

Like a moulin rouge admin?

You got me there :D

Time for them to hit the gym and call a lawyer!

Yeah, that's what I was wondering. Strange that there is a SPOF for Facebook/Insta/Whatsapp.

I had the impression that they are still run like mostly independent companies who only share some data on the backend.

AFAIK, I remember reading something about Instagram moving into Facebook datacentres some time ago. I believe they were on AWS before the acquisition.

Yeah that's what I was thinking too, so this is probably DNS related.

Might I ask what’s SPOF?

Single Point of Failure.

Thanks

Single Point Of Failure

Reply on news.ycombinator.com