Don't use Cloudflares 1.1.1.1 on servers | Hacker News Reader

Don't use Cloudflares 1.1.1.1 on servers

5 pointstmikaeld8 comments

We've gotten rate-limited out of the blue on clustered development servers in the past 3 years now, this last one was on servers we setup 830 days ago, before we knew that getting rate-limited/banned on DNS servers where even possible. The worst thing about the last incident was that we entered a death spiral, DNS resolution failing started a logging job, that failed (due to DNS resolution failing to call log server) that then started a job about the failing DNS resolution.. You get the gist..

Of course, this is an issue of engineering and code, not only a rate-limiting issue.

However, many developers rely and depend on upstream DNS resolution to "Just Work" when you add it to a server, which has been the case with Googles DNS servers for the past 15+ years that I've been a sysop. I'm just hoping that this time, this will get SOME attention, because either you want dev-ops to use Cloudflare DNS on servers or you don't - and if you don't - there should be an official warning that this WILL happen, you WILL get rate-limited eventually.

> However, many developers rely and depend on root DNS resolution to "Just Work" when you add it to a server

As a sysops you're probably aware that neither Google nor CloudFlare are DNS root servers.

Using actual root servers through your own resolver would have avoided this issue. Bind doesn't even need any config for that use case.

Of course, it depends on the use-case, what I meant was "upstream DNS". I've edited.

Adding to this data-center servers should at very least use a proper set of caching DNS servers at the edge of their network and those should talk to the root servers as to not add to the global abuse of the anycast clusters. I've seen some companies go as far as to run Unbound on each and every server to improve the retry and caching mechanisms to great success. Unbound can also raise the min TTL as some applications get quite abusive if they use really low TTL's and the applications are making requests for every action. i.e. raising every TTL up to at least 30 seconds if they were lower. The excessive retries can compound really fast especially when applications and systems are not properly configured which is often the case. If people are not sure what I mean, run a UDP capture at the edge of your network and one may find that for each request an application makes there could be as many as 12 DNS requests. It adds up very fast.

    App1 Unbound -> Data-center edge Unbound instances [1-4] -> Root DNS  Anycast clusters

Unbound can be configured to either pick the fastest resolvers or to cycle through all of them and retry in the background when one fails to re-select it when it starts resolving again. This avoids a lot of outages otherwise known as "It's always DNS".

But when setting up a full recursive resolver, you should avoid using root servers directly for queries, but rather mirror the root zone locally:

https://datatracker.ietf.org/doc/html/rfc8806

What kind of volume was this? I have a server that does some rather specific DNS monitoring resulting in millions of unique lookups with 1.1.1.1 a day.

That's the frustrating part of this and the inconsistency, we're doing benchmarks one day, making thousands of lookups, adding/removing domains, then during normal day operations we're getting blocked.

Is this only DNS or have issues with accessing CF networks? Do you own the subnet the server is on is it shared with others? Wondering if this is because of other traffic from the subnet and also affects you.

These are on spread out external IPs (VPSs) so not within CF networks or specific IP subnets. The common denominator is that at certain bursts of traffic, we get blocked.

If this had some kind of pattern we could avoid or improve, I wouldn't even bring it up.

Reply on news.ycombinator.com