> We saw an unusual spike of activity that began on September 14, 2018, and we started an investigation. On September 25, we determined this was actually an attack and identified the vulnerability.
I'm unclear why it took 11-12 days to uncover this?
I'm unclear why it took 11-12 days to uncover this?
Someone noticed a spike on Sep 14th which was a Friday. Person A probably sent an email to Person B asking for information. Person B finally replied to ask person C on Monday. Person C gets looped in Tuesday and assigns Person D to take a look at it. Person D is finishing up a task and starts looking on Friday. Everyone thinks this is fine as the issue is probably just a front-end bug that's causing looped requests. It's not considered a security issue yet.
They make a little progress on Friday, but are out of town in Tahoe for the weekend. Monday the 24th rolls around and they finally figure out that it might be an attack and report back up the chain, a process which took 24 hours. Now it's the 25th, which is 11 days later.
1-Mark opens calculator
2-gets how much $$$$$$$$ he’ll lose if fb shutdowns for couple hours
3-lets hack continue
4-profits
Joke aside, i’d estimate they have about in the ballpark of over 200 security engineers that have alerts for when smth like unusual activity happens
Second, if the counters that were being triggered by the attack weren't obviously malicious, it's likely that for some amount of time it was treated as a lower priority figure out what's going on deal rather than a sign of something being critically wrong.
"Hey, we're getting 25% increase in access to the video uploader but the number of successful uploads is normal. Did we break the page somehow and people are having to reload the page?". And then somebody spends a day looking at what was pushed the previous Thursday before deciding that the widget is working correctly and escalating to someone else.
It's also highly likely that the product teams at Facebook (who would probably have started the investigation, since it was product-related counters going up rather than something obviously security-related) might not have sufficient access to logs to determine the exact sequence of events. Maybe they didn't have the raw IP addresses of the requests, so couldn't realize that all this increase in traffic was coming from a handful of systems. Maybe they could only see the requests to their own service, but not the full path of requests to other services that the attacker used to pivot from one vulnerability to another.
A week after detection to deciding it was a security issue seems like a pretty plausible timeline to me.