Helping to Build Cloudflare, Part 2: The Worst Two Weeks
blog.cloudflare.com
blog.cloudflare.com
Particularly the value of getting recurring core dumps to zero, such that they stand out enough to find and fix. Reducing log/stats noise has been a traditional go-to for me.
Moving sshd to a non-default port, for example, initially sounds like useless security through obscurity. But, it then makes unusual ssh access stand out and get noted.
It just doesn't mean much. They had something like 2 weeks worth of logs. And further, they never explicitly stated that the logs had enough context to show if it was being exploited. That is, are there any fields in the log files that could distinguish normal uri accesses from malicious ones?
I'm curious if that's all deliberately careful wording because they know, that they "don't know".
It's true iff you haven't made any effort to find evidence whatsoever OR there's a zero percent chance that you would find evidence even if it existed. This is a very rare circumstance, and one that doesn't apply in this case.
There were some logs that. had there been exploitation, had a non-zero change of providing evidence of said exploitation. They looked in the logs, and there's a non-zero chance that, if that evidence were in those logs, they would have seen it.
P(exploitation) < P(exploitation|they had and looked at the logs)
That's the definition of evidence.
https://blog.cloudflare.com/quantifying-the-impact-of-cloudb...
"We have the logs of 1% of all requests going through Cloudflare from 8 February 2017 up to 18 February 2017 (when the vulnerability was patched)...Requests prior to 8 February 2017 had already been deleted."
One of the most interesting things we had was the core dumps. Randomnly (depending on memory state) we'd crash rather than dump out memory in the HTTP response. We had all that data going back over the entire period. That gave us a lot of confidence this hadn't been exploited because we could see the rate of crashes plus we could see the actual core dumps to see the memory state when the crash happened.
That leaves whatever the scale of passively trolling, say, Google's cache might have been. Unknown, but probably not huge.
Starts at 25:30 for anyone curious.
> We were recording the core dumps [...] We didn't look at this
Starts at 25:30 for anyone curious.
> We were recording the core dumps [...] We didn't look at this
Admittedly, this isn't the kind of stuff I normally write, so I might be missing something. But since the ramifications of these memory leaks are so severe, I would think the industry would be adapting.