Hacker News Reader

Comment by coderholic | Hacker News Reader

https://host.io - an API to get domain name metadata (scraped web content, backlinks, redirects, dns data, ranking information and more). See https://host.io/google.com for an example.

We've been building out the infrastructure for a while (scrape all domains monthly, resolve all domains, progress the data etc), but have only recently launched the API (see htttps://host.io/docs).

We'll soon be releasing a Top 10M Ranked Domain list, like the Alexa top 1M, but 10M, and based on our own ranking signals, instead of traffic data like Alexa.

If you've got any interesting use cases for our data or any feedback I'd love to hear it! ben@ipinfo.io

View on HN

That's interesting, especially this, I think:

> Shared IP Address

> There are 68,258 domains hosted on 216.58.217.36 (AS15169 Google LLC) including google.com. Here's a sample: [...]

You wrote:

> scrape all domains monthly, resolve all domains

what does "all domains" mean? All domains worldwide? Or some sub set?

As mentioned, I think your project sounds interesting — still I'm wondering, how did you verify there's an interest for what you're building? (I'd think/guess there is, just wondering if/how you verified this, or knew beforehand)

All domains is literally all domains :)

How does one get a list of all domains?

Rapid7's project sonar forward lookup database is a good start. https://opendata.rapid7.com/

Some corps collect DNS logs from the infrastructure for domains that have been registered very recently.

UPDATE: The free tier is static at 1k and then you have to buy credits or it is 1k / month?

Currently the API is priced at $0.01 per call, with $10 of free credit when you signup.

We'll likely add monthly subscriptions in the future perhaps with a recurring monthly free tier, but we don't have them yet.

If you do run out of the free $10 credit and want more time to see if the API is going to be useful for you shoot me an email and I'll be happy to issue more credit.

Thanks

Where are you getting the link data from? Common crawl? 3rd party?

Same question for ranking info, assume you're not scraping yourself and using some like D4SEO?

We're doing the scraping ourselves. Just a single page per domain currently (so it's just links on the homepage), but we'll expand to a few pages per domain in the future.

We're also doing the ranking ourselves. We bootstrapped it by doing a regression against Alexa and other top 1m lists using a bunch of our own features (like number of backlinks, some content features, some IP address related features etc). We'll be sharing more details around this soon.

> If you've got any interesting use cases

What are some cases you know about this far?

Cybersecurity research / investigation (example here: https://twitter.com/host/status/1215709472066170880)

Intellectual property enforcement

Malicious traffic / account creation investigation

Domain name or competitor research

Market analysis (eg. find out how many customers publicly traded hosting providers have)

Reply on news.ycombinator.com