Hacker News Reader

Hey, we're still kind of new and kicking the tires. Any UX feedback or data feedback is appreciated!

It's pretty much missing or empty data for the entire .au TLD (e.g. even google.com.au is not present).

Living here, I know that their whois server rate limits heavily and the ccTLD zonefile is not available, so I'm guessing those are probably contributing reasons. Do you guys do any crawling at all?

I think this is a fantastic concept though. Knowing that whois database and zonefile access is often protected for commercial motivations, it really irks me. Open it all up.

right now we're getting .au domains from Open Crawl. It's tricky because the zone file is not available like you mentioned. If anyone has any ideas on how to get more .au domains, we'll gladly implement.

There's some reverse engineered zone files for a lot of cctlds at http://viewdns.info/data/.

CT logs come to mind, if you aren't using them already.

DNS queries, if you have access to recursive resolvers.

we're using CT logs for getting hostnames (you can see it at the top of the page of any domain).

We have a great recursive resolver source but haven't been able to integrate it into the data pipeline yet.

thanks for the note and ideas!

Reply on news.ycombinator.com