Does anyone have a link to a blog that explains why I can't have every DNS entry in a database at home?
I don't want a 'you could never get this 1%" because of X straight up. What percentage could I get and why is this not easy?
I don't want a 'you could never get this 1%" because of X straight up. What percentage could I get and why is this not easy?
First, grab the root zone file which points to all the TLD nameservers: https://www.internic.net/domain/root.zone
Next, get zone files for the TLDs themselves. Not every TLD makes that freely or easily available, but https://github.com/jschauma/tld-zoneinfo is an excellent resource.
The author of the above repo runs https://netmeister.org/tldstats/ which will give you an idea of scale; they're able to get 98.89% of all TLDs.
(Correction: zone files for ccTLDs are mostly unavailable. That 98.89% is only domain count coverage, not zone file coverage.)
But mostly it's about how the DNS changes, a lot. Sites move, geo records, etc. Getting those changes to every household would be a lot of traffic.
Before DNS folk were sharing a common /etc/hosts file - and quickly determined there had to be a better way.
So, DNS with a dozen root servers that stay fresh, caching down stream.
2) The TTL for a DNS record can be 0.
Edit:
3) AXFR is blocked on 'most' zones. You would need this to know all the records in a zone.
When Jane Deer creates a new Blog jane-deer.com, your local database would not know which server to contact to resolve it, and nobody is gonna notify the database in your home.
So you would have to contact some other kind of database outside your home to find out how to resolve that domain--which you might have already realized sounds a lot like a (non-local) DNS server...
Beyond that DNS servers are not just a "database" of records, they are services that can return different results depending on who queried. And this is quite common with CDNs.
There are many more reasons, but let's talk about what you could do.
You could download a list of all registered domains, and then query all of them for the most common records. It would take hours, not include 99.9 percent all records, and be out of date the second it completes. With this database, you could visit some websites, but many host services on subdomains and you have no way to dynamically get that list when you're populating the database.
The next question from there might be why do DNS responses change so frequently? I don't have an exact statistic, but most of them don't (I would guess 90%), so you could probably indefinitely cache those. But the ones that do, iirc, it's usually to allow quicker changes (esp. if the site is under active development), load balancing, global distribution, etc.
I hope that's helpful.
IPs change out from under domains constantly. How would knowledge of those changes make their way from the owner of the domain to your local database?
If you'd want a single source of truth to receive and distribute that info, that's a single point of failure and comes with lots of technical challenges. And it's less "you could never get this 1%" and more "you will never need 99.9% of this"
------------------------------------------------
However, that's all entirely unrelated to this Cloudflare incident, which involved no domains at all, only raw IP addresses. This was an incident involving BGP, which is how the various providers of internet backbone know which other provider they connect to is able to answer requests made to a given IP address, and how changes to that information propagate.