From side project to 250M daily requests | Hacker News Reader

From side project to 250M daily requests

medium.com

87 pointscoderholic15 comments

This is information I would never accept an external dependency on. There are not too many IP ranges around, so this information can easily be stored somewhere and used without the need to query an external service.

Having the data locally allows for potentially synchronous lookups, or at least for lookups with an availability guarantee which makes your code much simpler to reason about.

Plus, you are guaranteed to get the information, independent of the availability of a third party.

And finally for $100 per month I get access to a weekly updated locally available database (that's what Maxmind charges for the city-level database) no matter how many queries I'm going to issue locally.

Flat-fee access to a local database would even allow to bulk post-process web server log files (yeah - they still exist and don't come with the usual privacy issues surrounding third-party analytics providers) within a reasonable time- and cost frame.

Not everything that can be an external service has to be an external service and for geolocation I definitely cannot see any advantage to not having this data stored locally.

This isn't big-data and will easily fit any amount of RAM (if it even needs to), this doesn't require a costly sys-admin team, this doesn't require any hardware knowledge. This is about fetching a file and putting it somewhere on the server. You are doing this daily with your daily web-browsing.

Now, to OP, I'm very happy for you and I appreciate the service you are offering and I'm very happy that you are solving an issue some people are having. I don't want to belittle this at all.

I'm just saying that while I might personally err a bit too much on "doing it on my own", I absolutely cannot see any justification to do IP geolocation with an external dependency.

I could see the value of having this as a docker container to run internally as a microservice, but agree I would never rely on an external service for this.

We've done this at Clever, if you're interested: https://github.com/Clever/resolve-ip

We use the same dataset that IP info started with (from MaxMind), although it sounds like they have many different sources now. It's worked fine for us, matches over 99% of the IP addresses we look up, and generally responds in less than 5 milliseconds - basically only limited by network latency.

Right now this is published to a private Docker hub repository, for no good reason, but if people would find it useful we can make it public.

Sometimes the reason things succeed is simply because they are simple. Simple to understand. Simple to use. Simple to integrate. Simple to pay for. Just simple. One Feature, maybe. Three max.

Good insight into the engineering that went into this. Functionally, is this "MaxMind-GeoIP-as-a-service"? Any other data sources?

http://dev.maxmind.com/geoip/

Yeah, lots of custom data. See https://ipinfo.io/data for an overview

Since you are building off of GeoLite which has a ShareAlike license (http://creativecommons.org/licenses/by-sa/3.0/), does that mean you are distributing your modified database under the same terms?

It's a good question. I've been in touch with Maxmind's legal team before and they were happy with what we were doing, but that was before we had started building out our custom data.

My thoughts are:

1) We're not modifying the maxmind DB. We download their MMDB file and leave it completely untouched. Our code basically does something like this:

    if ip in custom_db
        return custom_db[ip]
    else
        return maxmind_db[ip]

2) Our database is openly available via our API. Anyone can access it without needing to signup etc.

They're the principles I've been operating under, but I'm not a lawyer, and I'll definitely go consult with one to get a definitive answer on where I stand with this. If it turns out that we're not meeting the license terms then we'll certainly make whatever changes necessary to ensure we do.

The "Network data" is just ripe, arin etc data that is available publicly to anyone.

That's one of the raw sources that go into the network data. There's also the routing tables (without which you can't always match ASNs to netblocks), other datasets, and a bunch of custom scripts that post-process the data, performing cleanup, heuristic matching and more. And then putting it all into a data format that supports handling 250 million requests a day, and serving over 90% of them in less than 10ms.

So you use your own BGP tables or use HE or some other service to do it for you ? 250 mil requests is just ~2800 req/s. Serving 90% of them in 10 ms using api is not really any achievement, considering what kind of data you are serving. I've worked on api with 1 bil requests a day and 97% of them are below 10 ms (And the best is that only 2 machines are needed for that).

One service you could consider offering is "resolve IP at past date". I don't know that the standard providers do it and it useful when, for instance, batch-processing a backlog of session data.

I'm curious... are you seeing 99% using your free plan or are you actually selling the $400 plans? What is the conversion ratio?

This just placed me 1000km from where I actually am - the porn ads offering to meet local girls do much better :(

In Germany the only useful thing this site would give, would be the region, not quite good for dealer finders..

Reply on news.ycombinator.com