Hacker News Reader

Comment by chrissnell | Hacker News Reader

Apologies everyone for the somewhat o/t question here but, what do you guys do with your 50Gb of logs every second? Where do they go after they leave nginx?

View on HN

definitely interested in this. Where do you send this for analysis? Elastic search? Can that keep up?

NSA :)

Here's a blog post on exactly what we log:

http://blog.cloudflare.com/what-cloudflare-logs

TL;DR: we generate logs at the edge, those are turned into aggregates to display analytics data (e.g., page views, hits, bandwidth), then logs are discarded unless you're an Enterprise customer in which case we allow you to download the raw log data for 3 days.

To the NSA, presumably: http://exiledonline.com/read-yasha-levines-introduction-to-t...

To be 100% clear: we have never been asked or ordered to share log data with the NSA. We've not participated in any program like PRISM and would fight vigorously if we ever were. We do receive law enforcement requests on occasion, typically to determine who owns or hosts a site behind our network. When we've received law enforcement requests for customer data (e.g., account information like the email address of an account) which we determine are abusive or do not follow the principles of Due Process we have and will continue to go to court to fight for the rights of our users. Whenever possible, even if the legal request meets our standards, we also notify customers of legal requests and allow them to challenge the requests themselves before turning over any data. We take this extremely seriously and spend significant technical, legal, and public policy resources to ensure law enforcement's job is neither easier nor harder by the mere existence of CloudFlare.

Sure, but why should I have any reason to believe you?

If you had been served with a national security letter, you would be obligated to lie about it.

Plus, you have a pretty significant financial incentive to lie about how great Cloudflare is, with no downside, since you're not under oath on HN. And even if you were, officials who have lied about the extent of surveillance programs while under oath haven't been prosecuted.

Representatives of Facebook lied. Ditto with Google. Etc, etc, etc.

No. National security letters can act as gag orders, obligating you not to talk about their existence but you're never required to lie and say you have never been served a letter.

In that case, you can be cagy (as Google has I believe) and say something on the lines of "I can neither confirm nor deny receiving a NS letter" which of course is double speak for "I have one, and I can't talk about it directly".

Not a customer of Cloudflare, but that seems a bit unlikely given the CEO has called NSA gag orders 'insane':

http://www.washingtonpost.com/blogs/the-switch/wp/2013/09/12...

A CEO disagreeing with gag orders has not been a good indicator in the past.

For most customer they are stored just long enough to provide support, create aggregates for the analytics dashboard, and update threat profiles. Our enterprise customers have the option collect raw logs through SFTP for up to 3 days.

Reply on news.ycombinator.com