Hacker News Reader

Comment by wielebny | Hacker News Reader

So, the traffic is being sniffed and analyzed, and that service was advertised as privacy-oriented?

"The service" is the DNS service. If you send random garbage to random IP addresses I think you waive the right to privacy.

>APNIC gets to see the noise as well as the DNS traffic

>Huston emphasised that APNIC intends to protect users' privacy. "DNS is remarkably informative about what users do, if you inspect it closely, and none of us are interested in doing that," he said.

Maybe it is reasonable to take them at their word as they seem trustworthy, but we should at least consider the fact that at least some of this DNS traffic is indeed being analyzed.

No, wait.

Users of the DNS service get the privacy guarantee.

Non-users do not. If you floodping 1.1.1.1 you are not a user of the DNS service and the privacy terms don't apply to you. Rather you're a member of the Misconfiguration Club, and the site you're pinging has the usual right to analyse your pings.

What if somebody has a bad DNS resolver and what he qualifies as a valid DNS request, researchers do not.

I get the general idea, but having "user-privacy oriented" and "we collect everything and make it available to many researchers" services under the same IP may lead to some issues.

Even a bad DNS resolver will still send to port 53. The privacy policy probably applies to anything on ports 53, 80, and 443.

and the DNS over TLS port, 853

Oh, in that case you can apply those issues to all of Cloudflare. They serve many thousands of websites from each node. God only knows how many different privacy policies may apply depending on which bytes you send to TCP port 80.

I'm pretty sure that all the traffic is being analyzed. The only thing they publicly committed to is not saving your Ip address.

> "Under the terms of a cooperative agreement, APNIC will have limited access to query the transaction data for the purpose of conducting research related to the operation of the DNS system."

https://developers.cloudflare.com/1.1.1.1/commitment-to-priv...

DNS traffic, no. Random garbage traffic misdirected to 1.1.1.1, APNIC is studying.

Does all this garbage traffic affect the performance of Cloudflare's servers? There must be some cost (performance and $$$) to filter this traffic. Was that a consideration when deciding whether to use 1.1.1.1 instead of some other IP address? :)

Vendors like Cloudflare have usually really easy to use and cheap measures to drop traffic otherwise they would not be able to provide DDOS protection. Usually they have BGP or other means to propagate blacklisted IP ranges to peers meaning that not even their peers will route the garbage towards them. This is how you can survive DDOS that is bigger than your pipe while serving legit traffic (coming from different ranges than the ones you blacklisted).

No. We have a lot of capacity. A lot.

For ordinary singlehomed users who don't get the "a lot". As an example cloudflare has 40Gbps of capacity to the SIX in Seattle. I would guess that they also have direct, at minimum, 10Gbps PNI peering sessions with other huge ISPs in the Pacific Northwest which never see the SIX fabric. So probably add another 20 individual 10GbE circuits at bare minimum to that 40 figure. All of which helps spread the traffic load out rather than shoving it all down a few pipes.

https://peeringdb.com/net/4224

that's just the public stuff!

For huge entities like this and other top 5 CDNs it makes me wonder how many full time staff positions are dedicated to buying rack space, power and crossconnect in major colo facilities worldwide. How many contract law experts, telecom real estate analysts, etc. That before you even get into things like experts in Japanese contract law. Just for rack, power and facilities at layer one of the osi model before any networking happens.

I don’t know the exact number, but you’re off by about an order of magnitude roughly. Cloudflare peering is in the terabits/sec range globally.

At many non profit IXes the interface size to the fabric is public data and published by both the IX and on peeringdb. Such as at the SIX. They have not yet upgraded to 1x100GbE.

Another regional example, they have 20Gbps to the VANIX.

What is opaque is the size and scale of their PNI peering, which parties generally don't share. For example in a mid sized city where Comcast is the cable monopoly they almost certainly use a 100GbE interface direct to Comcast for just that isp.

Yes the scale is terabits globally. But it is highly decentralized.

> But it is highly decentralized

Isn’t that the entire point of a CDN, to have decentralized POPs scattered globally?

Yes, they may max out at 100gb per public IX in most cases, but they still have lots of 100gb peers all over the globe.

Yes, it's exactly the point. What I was saying is I am not off by an order of magnitude, I know exactly how big they are. Two ASes I do engineering work for peer directly with cloudflare.

> I am not off by an order of magnitude, I know exactly how big they are.

Your post implied a total of 200-400gb, the real number is 10x that (otherwise known as “order of magnitude”). I’m not disputing your knowledge or experience, but the post as written has issues.

no, my post was quite specifically about a single IX point in one geographical location, and was accurate. You confused me talking about interface sizes and counts at one IX point with cloudflare as a whole.

What difference does it how much they have as peering in one specific geo? Are you going to posit then that that is their capacity in that geo? I hope not, as Cloudflare has their own backbone and could easily and efficiently exchange traffic outside the geo depending on peering agreements.

Cloudflare is anycast announcing the space from something like 30 to 50 unique POPs worldwide, so the volume of shit traffic is significantly decentralized. It's not like 40 Gbps is hitting one location.

They scaled up recently and now have ~150 POPs. They specialised on DDOS defense from the start so traffic volume isn't an issue for them. Everything below 100gbit/s globally will probably be well within normal fluctuations. But I guess they talk to ASNs that send a lot of garbage and ask them to investigate on their side.

Sure, but they could also study DNS traffic if they wanted to. Or at least, with Cloudflare's cooperation.

If you want privacy, you never do DNS queries from an ISP-assigned IP address. Tor exits do DNS queries on behalf of clients. Decent VPN services also handle DNS queries for clients.

They don’t get raw DNS traffic. Ever.

But that's just a Cloudflare policy, isn't it?

Or are you arguing that even Cloudflare couldn't get raw DNS traffic?

That’s our policy and we’ve hired outside auditors to ensure we’re honoring it. If you have suggestions of what else we can do to prove we’re a company of our word, LMK.

Oh, I didn't realize that you're a CloudFlare cofounder.

I don't mean to question CloudFlare's integrity.

It's just that, for claims about privacy, I'd rather depend on more than trusting any one party.

Great. Use 1.1.1.1 and 8.8.8.8. You don’t depend on one party now?

Yes, thanks.

Even if you just use 1.1.1.1 you know you have 2 parties, cloudflare and their auditor.

There is that argument. But both are likely vulnerable to coercion from some adversaries. And of course, that's always a risk.

Will these auditors guarantee that you won't wake up one morning after a troublesome sleep and start monitoring this to protect us against Nazis? Because that might be good to ensure we believe your word.

I have a problem with these comments because they're basically a false dichotomy. No, CloudFlare can't ensure they play fair. Can your ISP? Who can?

Because these sorts of comments read to me as "yeah, you're the best right now, but are you perfect? No.". Nobody claimed perfection, and there's value in being the best.

Precisely. There's no perfect. OK, so I use IVPN. I've known a principal for many years. I write stuff for them. And I trust them to protect my privacy. More than any other VPN service. I also trust a few others, almost as much. And way more than I trust my ISP.

But I never depend on any one of them. I use nested chains. That's my no means perfect, because routes are relatively static, unlike Tor with its frequently changing circuits. But the basic idea is the same. Compromise would depend on collusion, perhaps forced, of multiple parties. Or some serious traffic analysis.

Here, there's CloudFlare, its auditors, and perhaps Google. So maybe there is distributed trust in that. But still, I'm happier to put more independent parties between me and them. Tor, or at least a nested VPN chain.

Cloudfares CEO was guaranteeing that they have an external auditors to ensure the company keeps its word. I'm asking whether the same auditors can ensure that the CEO himself won't decide one morning to go against this word wrt this topic. In the same way that a ToS might state "we will never sell your data to third parties" and a CEO can't break that promise, what guarantee beyond "some external people are looking at what we do" and "oh, just trust me" do we actually have? Im asking if the auditors will also guarantee the CEOs word.

This is relevant as the CEO has previously woken up one morning after a troublesome sleep and it has been argued, gone against his word (for good and anti nazi reasons). Many argue that he was entirely within his right to do so, and he was! So in this case, would he be entirely within his right to start monitoring all that data. As he asks, paraphrasing: "what more can I do to ensure my word is good."

I'd bet that if they one day drop that promise from their page they'll quickly hit the HN front page. If you don't trust, just monitor their policy page for diffs.

Will they at least get processed information about the DNS traffic (like how much of it is normal recursive queries versus broken garbage which happened to go to port 53)?

It was reworded enough times to make their promise vague and not well defined.

I’m Cloudflare’s CEO. What questions do you have?

I’ll start: do we ever store 1.1.1.1’s users’ IPs? No. They’re never written to disk. And APNIC never has access to them.

What data do you provide to APNIC? We give APNIC reports on non-DNS data that’s hitting 1.1.1.1. It includes information like: what protocols are sending data to the IP, what’s the volume, where it it coming from?

For DNS users of 1.1.1.1, we never provide APNIC any identifying information. We don’t upload any data to them. While they can query for questions like: “How many queries came from India in the last 24 hours?” they can’t query anything on a specific user.

If you have concerns, ask them here. I’ll answer.

Are you aware that your public resolvers are actively breaking DNS-based GeoIP (striping EDNS0-ECN and not using source IPs geo-localized as the requester would be)? and if so, what is the rationale for it?

Yeah I tested it out and switched back, it made performance to Twitch in particular quite bad for me. I don't get that issue with Google DNS though.

Google's take good care of it... and they explain precisely what they do on the topic...

https://developers.google.com/speed/public-dns/docs/ecs

https://developers.google.com/speed/public-dns/faq#locations (when EDNS0/ECS isn't supported)

The tin foil hat brigade might suggest that this is deliberate to ensure that only what's served by cloudflare gets to be fast...

Why is healthy speculation now considered "tin foil hat brigade"?

Conspiracy theory isn't a dirty word.

Speculation keeps people informed and alive.

I think everyone's concern should be that Cloudflare or Google might just replace each IP in their 24h logs with a random UUID and call it anonymized. Both companies can potentially store enough information to correlate DNS requests with regular traffic logs.

It doesn't take much guessing to know who sent an anonymous DNS request for example.com to one of your countless PoPs if your CDN logs a HTTP GET request to www.example.com at the same location a few milliseconds later.

We're not storing the source IP addresses in any form. Not raw, not "transformed", not anonymized, not hashed. They are not being stored.

Our business is not about tracking people; it's about selling our service to businesses to make their web sites/APIs/applications faster and more secure.

From https://developers.cloudflare.com/1.1.1.1/commitment-to-priv...:

"Specifically, APNIC will be permitted to access query names, query types, resolver location and other metadata via a Cloudflare API, that will allow APNIC to study topics like the volume of DDoS attacks launched on the Internet and adoption of IPv6."

I interpret "query names" as some values obtained from DNS queries hitting 1.1.1.1, e.g. "foo.example.com".

Is your answer to "What data do you provide to APNIC?" complete in the statement above?

Thanks for clarifying.

No querying IPs, ever. Nothing personally identifiable. APNIC can query things like: how many DNS queries come from the UK? How many query for google.com?

Hello,

Are the gigabytes of junk billions of tiny requests or are there large requests as well?

Are you finding it more difficult than expected to manage the data?

I'm a 1.1.1.1 customer since you launched, thanks a lot for it.

Nope. We have a lot of excess capacity. Doesn’t increase our costs. But, that’s a longer conversation…

Neteng here: without seeing the traffic charts for individual interfaces and aggregations, I would bet cloudflare has a shitload of excess inbound capacity. They are a content pushing CDN. I would bet that at a major IX where they have one 100Gbps port that their out:in ratio is 90:10 or greater. So if they only have 6-9 Gbps of traffic inbound on a 100GbE port to a fabric consisting of 80+ bgp peers, they have a lot of extra capacity to absorb unwanted inbound traffic before it becomes an operational concern.

Have seen edge traffic charts for major porn hosting companies and the out:in traffic ratio is like 97:3

How do you know where the requests came from without IP addresses? Do you log ASN or something?

We keep IPs in memory to help stop abuse and debugging ng other issue, but we promise to purge all logs within 24 hours or less.

>they can’t query anything on a specific user.

What exactly do you mean by "user"? Can they query DNS traffic by IP address / subnet? Exactly what are all of the restrictions there?

EDIT: Is there a whitelist of things they can query by or do you simply trust them to be good citizens, have a binding legal agreement, all of the above?

No. We have a legally binding agreement. And, more importantly, we don’t store or give them access to IPs or anything else that may be associated with any individual. Look at a DNS query, look at what could be identifying — let us know where concerns are. My hunch is we’ve thought of it. If not, we will fix. We don’t want personally identifiably info. It creates a legal risk for us. We purge it as quickly as we can.

We don’t want personally identifiably info. It creates a legal risk for us. We purge it as quickly as we can.

Ding ding ding, we have a winner. If more people would realize this, we would have less data breaches. To get there, a data breach must become more costly for the companies.

Is there personal data in the domain names queried, as opposed to the IP addresses querying? I think there probably is. For example, consider the bug in iTerm2 that ran a dns query for any text cmd-clicked on.

To protect against that, could you commit not to log to disk any queries that come from fewer than N ips in 24 hours, and not to expose rare queries to APNIC or internally? (Not a demand, just brainstorming mitigation.)

It might also be fun to give an internal team access to the data you consider safe, and challenge them to dig up personal data from it.

Do you plan to publish usage statistics over the coming weeks? I'd be keen to know what kind of (valid DNS) volumes you've seen since the announcement. And if there are any obvious patterns of ASNs or countries using it a lot (unless you don't want to disclose that to avoid being blocked there).

> I’ll start: do we ever store 1.1.1.1’s users’ IPs? No. They’re never written to disk.

Is there a guarantee that this will always be the case? Might there, in theory, be a point in the future where users' IPs are collected and stored?

Loving 1.1.1.1, btw.

Does APNIC get a sample of the raw packets of non-DNS and non-HTTP/HTTPS/QUIC protocols, so they can figure out what they actually are and why they're being sent towards that network?

I expect that no human from APNIC or Cloudflare will ever look at raw data from 1.1.1.1, nor will any of it be recorded, or used in aggregated data that retains any personal information.

As personal information count full IP addresses, the content of requests, or any set of data that can be used to recover these.

That is what "we respect your privacy" means.

Neither we nor APNIC can query “what” or “how many” requests from any IP have been made.

We can query things like:

1. How much query traffic is from Africa? 2. What’s the peak time of query traffic? 3. What are the most popular DNS authoritative servers?

If you have specific concerns, please raise them here.

What's in it for you guys? How do you make money off of 1.1.1.1? Thanks!

Brand. How much would you pay if you were us to associate your brand with privacy/security and speed?

Performance. Our core business is making our customers fast and safe. More people using 1.1.1.1 means our Authoritative DNS service inherently faster for anyone who uses it.

Recruiting. Our mission is to help build a better Internet. Lots of places the people on our team can work. That they work for us is often because employees believe in our mission. 1.1.1.1 helps with that.

> Our mission is to help build a better Internet.

I've been working a lot with open data and I have huge problem with Cloudflare ruining open internet with bot protection and such. The issue I have is that public data is public, be it bot or human.

I'm having real issue with you guys saying that your mission is to better the internet when you break shitton of floss apps that are essentially harmless and people who want to do harm, crawl at huge rates for commercial purposes break your systems like it's made of twigs. I'm saying this as a person who works on both sides and can't help but call you out.

So sorry unless you turn to non-profit I'm really not buying your "helping the internet" song.

I agree. Part of what makes the web the web is that anyone has to be able to crawl its data, and build new technology on top.

As long as Cloudflare blocks that, it's hostile.

Maybe a way to register as spider with Cloudflare, and get a token that one passes in the header, with strict rate limits would be much better than the current solution of just showing captchas.

I guess they don't, they just make a better internet, which helps their customers of their other services, and the rest of us. Seems like so good old fashioned altruism to me.

Reply on news.ycombinator.com