I started hacking away yesterday at making an alternative database that seems to be more accurate, but it's still in the works. Who here would be interested in a free alternative, including and excluding cities, updated on a daily basis, and if so, which format's would you like to see?
As far as formats go, if anyone is thinking of putting together a custom database with information keyed on IP address or IP ranges, I'd encourage you to look into our MaxMind DB format:
http://maxmind.github.io/MaxMind-DB/
There are open source file readers available in many languages: https://github.com/maxmind/?utf8=%E2%9C%93&query=MaxMind-DB
There's also an open source writer https://metacpan.org/pod/MaxMind::DB::Writer
Our officially supported libraries are actively maintained and pull requests are always welcome.
I'm actually about to begin writing a blog post on how you can create your own custom databases. I'll share it on HN once it's ready.
A couple years ago I worked on a project that involved processing the "Internet Census of 2012" data.
The MaxMind DB format was a fast, efficient way to store simple data keyed on IPs and subnets.
https://www.maxmind.com/en/geoip2-city-database-accuracy
To get the actual product comparisons, you'll need to enter a country in the input box. For instance:
https://www.maxmind.com/en/geoip2-city-database-accuracy?cou...
You can try refining the distances to see how the different products compare. If you fiddle with it a bit, you'll see that the results do vary from country to country. So, "good enough" will depend on some combination of your tolerance for error + which countries you care about.
A free service like locationkit (seen here a few days ago IIRC) looks OK, and I'm reasonably sure it's supported by revenues for other services from the parent company. However, even that feels like it might get shut down if it becomes too big a burden.
Something with a good understandable license that I can pay a reasonable amount ($5-$20/year, for most low-mid volume use cases) would be great, as long as I can really use it for commercial purposes. I had some high hopes for Yahoo services a few years back, but they seemed determined to plague everything with confusing or commercial-hostile licensing.
That seems to be way too low.
$20/month+ could make sense, but less than that should be simply free (limited edition).
Dealing with payments is just too much overhead.
With maxmind, $10/year might be all that I use (considering 1000 calls might only be $1 charge).
Should work too.
I would personally not want to pay up front without knowing anything about this database from a company (or private person) that only asks 5 bucks a year for it.
Seems like anyone who can look at BGP tables and whois information can get more current and accurate data than MaxMind.
The biggest one is just routing information and rDNS hosts which give away a lot of the information required anyway. If you were really thrifty you would make agreements with services (or use services you run yourself) that have users supply their location information and use that to construct a more granular database. I suspect this already happens to a degree, some of the fake information I give out has ended up in at least one IP address database.
I strongly considered doing this at one point but lacked the connections to pull it off properly (not enough sources of data I could get without approaching random people, basically). A secondary concept would just be to crowd source it and have the information freely available, but I was never sure how to deal with people poisoning the database or just honestly supplying conflicting information.
If I could I would redesign internet routing in a way that the webpage doesn't know where a request comes from.
https://www.gov.uk/government/publications/vat-supplying-dig...
"the business making the supply must obtain and keep 2 pieces of non-contradictory information to support and evidence the member state where the customer is normally located."
One of those being: "the Internet Protocol (IP) address of the device used by the customer"
So sign up for something while abroad and you could get caught out.
Estimating my taxes, shipping charges and delivery date are great too. Even sorting products and choices by what those in my area prefer is great too.
The geoip doesn't need to be street level accurate, but to the nearest ~30 miles can make the online experience much more pleasant and sensible.
"some licensing foo" is a great way of eliding that in some countries, the service would be illegal. The alternative is that the product could not exist at all, if there is no way to negotiate a contract for it.
You can certainly argue that products should not exist if that would involve contradicting some abstract ideal for society, or that it's good for things on the internet to blatantly violate mundane real-world laws until real-world governments become irrelevant. Respectable people hold both of those positions. But they're not very widely-accepted positions.
I'm not being pedantic about the argument, I just don't see how there's a benefit to any fundamental values if we still allow this delta, but we're picky about what technical means are used to make it happen. I could see an argument that geo-IP databases are often wrong and that's a problem, but that seems like a very different argument.
The internet already has that - it's called spoofing a packet.
But in all seriousness - this is an extremely bad idea. Having a unknown source would make it extremely easy for horrible people to do bad things to good people.
Really? Do you want everyone on this planet to get taxed equally too? Because if he answer is 'yes', then you're strategy isn't working.
- Free version: only updated once [every 6 months / a year] and non-commercial use only
- Paid version: $9.99 for a year of free updates
- Updates come both in diff and in full version (for the commercial version)
- Use a common format, or use multiple formats. People love csv, but more serious applications probably use an existing format (I see maxmind a lot in this thread).
- Perhaps an API with heavy ratelimiting for people who need only a very occasional lookup (for very occasional use, I use ip2location.com/demo currently; for more serious stuff I download a database).
Having a free version with a nice license might also get you included in a lot of open source software, which increases name visibility and makes people write parsers for your format of choice.
A minor note on payment: accept Paypal and/or Bitcoin, or I (and many other Europeans) can never pay you. These credit card-only services are impossible to order from.
Idea for the diff: if you have a binary format, perhaps use this: https://www.chromium.org/developers/design-documents/softwar...
Re: CC's, I'm also based in Europe (The Netherlands) and also hate services that only allow credit cards.
Thanks for the link regarding diffs, I'll definitely give it a read.
I refuse to have a paypal account, and recent changes to how they process payments mean I can't just pay with a cc without having an account.
If you are accepting payments, it might be a good idea to have more than one payment processor.
Especially if paypals notoriously badly timed account suspensions might force a existential crisis for your business.
Getting the person who has the card to enter the necessary card details to a web site is not always convenient. On the other hand passing the card number, cvc etc to many people in the company has certain risks associated with it. Using personal credit card is painful, especially with monthly fees, since you need to do all the paperwork to get money from company.
Usually the credit card invoices are not enough for the bookkeeping. You need to have more detailed record about what was purchased and about the VAT included in the price. With shared company credit cards somebody then ends up chasing the actual receipts, which happen to be in somebody's inbox.
In most cases it is just so much easier to handle things with invoicing. The invoices are handled electronically and usually companies have systems in place which allow the accounting department to just pass the incoming invoice to whoever for approval.
A quite good solution this would be a virtual credit card service where somebody could assign "virtual cards" to whoever on the company and put specific monthly/yearly limits on those. It would be pretty nice to add the collection of receipts to this one as well. So the person responsible for specific virtual card, would also upload the invoices that match to the payments made.
I have a JS library coming also for use when the w3c geolocation fails. The data is open and downloadable with cities etc (CSV right now). There's a mailing list for updates, and anyone can contribute just by opening the website in a browser!
I myself, could also be interested, but I think, I will leave that to my payment-provider ... (ugly, EU!)
Edit: Ok, I should not, but why the downvote, please? When I wrote something wrong, than please explain!
But, you (because you are so sure about the best method) by sure know, that the laws are explicitly for non-physical goods, where no delivery-address must be provided, right?
Here is a link, how one company is doing it (forget about government information, they seldom go so much in details!): https://rachelandrew.co.uk/archives/2014/12/08/consensus-and...
They use the IP address to validate the user given address and for later proof.
BTW: Downvoting all the time does not help the discussion. I try to discuss this topic seriously and I was bringing some very relevant topic here, and am constantly downvoted. (It is not my fault, that the EU came up with such laws, and provide no useful guide how to implement it!)
That "all" are doing it, does not guarantee, that one time a judge or tax controller will come and argue with you that you have no proofs at all. In this case, you are really bust, because the new law also says, that you are accountable for the tax to all the countries tax authorities. Much fun, when you get citations from different EU countries.
(Don't say it is not realistic: There are cases already public, where British shops got claims from remote EU countries, because of tax differences of <10 Euro).
And as long the EU does provide laws that are hard to implement in real life (only in the minds of the politicians they are simple) and nobody says how to implement them, it is better to collect all proof you can get.
https://www.gov.uk/government/publications/vat-supplying-dig...
There is clearly stated, that IP information is one of the proofs accepted for the country determination.
So, maybe next time, please first ask before judge.
There's huge demand for it - just search for geoip on Github. Some more pointers are here [0],[1]. So many projects, but most of them work with Maxmind data, which is the problem. Inconsistencies in the data they provide makes it painful to make performant fallback solutions. Performance across geolocations and lack thereof in "exotic" geos is another one. It seems none of these services is deployed across the globe. There's a need for a blazing fast, reliable service with granular, comprehensive data, that is performing well for millions of requests across the globe, be it for mobile or web traffic.
I found a great base[2] and I think Go is a perfect language for this kind of service.
Here are the free checkers I tested the other day, the IP is a good example for incomplete data:
checkers = [
'http://freegeoip.net/json/103.1.29.57',
'http://geoip.nekudo.com/api/103.1.29.57',
'http://ip-api.com/json/103.1.29.57',
'http://ipinfo.io/103.1.29.57/json',
'http://www.telize.com/geoip/103.1.29.57'
]
> It seems none of these services is deployed across the globe
Our free service has servers in USA, EU and Asia
> There's a need for a blazing fast, reliable service with granular, comprehensive data, that is performing well for millions of requests across the globe, be it for mobile or web traffic.
We have more than 2 billion requests per month!
I would be interested. Although a service like ip-api.com is not expensive ($8 per month or so if commercial use).
The structure of the data seems very simple: ip address range plus a location, I guess. So maybe just dump as a simple csv text file ? Always easy to process.
You have to make sure the legal side is covered: how do you get your changes ?
https://blog.mozilla.org/services/2015/03/06/combain-deal-he...
(I work at Mozilla and I used to work on the Location project. :)
I use MaxMind at work but because we're on the RoR stack, we have to download and commit the massive 30MB databases into our repository in order for it to be used on Heroku. We use https://github.com/yhirose/maxminddb which seems to be the only good option available that reads the new "mmdb" format. It's a complete pain to have to commit such massive DBs for it to work and would prefer for Rails to fetch the database upon initialization, cache it for at most 2 days and redownload if it hasn't already been restarted by Heroku.
If you can provide a solution to that, it would give us enough value for me to pay a few dollars per month for it.
It happens once per provision, and a cron job to keep it up to date.
But in specific answer to this question, if it's really public and free, why not see about adding it to Amazon's public datasets [1]? S3 is fast, durable, available, and I've enjoyed these other data sets.
Please don't try to use the dat file format. The binary files ending in ".dat" are the Legacy format, which has never been documented. Use the MMDB format linked by oalders in the top comment.
Can somebody make me understand why ipv4 assignments along with their requests are not made public?
What do you mean? With any of the RIR's you can get it down to a specific company. With ARIN, you're supposed to be able to get it down to a particular customer. What more information do you want?
But I understand that's not really an argument: you might say you don't want nukes to exist because they hurt people. I just wanted to mention it.
That said, maybe. Usually an rdns lookup does the trick already though (a4-8e-ef-78.dynamic.ziggo.nl or cpe-12-34-56-78.rr.com or 80-100-131-150.adsl.xs4all.nl is clear enough).
1) We can prompt mobile users to enable location/gps and get their location that way.
2) We can prompt modern browsers to allow location services and get their location that way.
This is if a user wants to have their location 'auto-set' instead of via pulldowns.
3) A hard problem to solve is - mapping a unified set of city names to timezones, and updating the timezones as per Daylight Savings Time and general timezone updates.
Anyone have references for 3) ?
Think it would be an interesting way to deliver services and data, just not sure how it turns into a business (since the API/service portion is designed to be zero profit).
How good is your global timezone data? Is it possible to map Google Places (from their places autocomplete form) to a timezone?
Let me know if interested or if we could maybe partner on this
To echo others, aim for low-priced and not free. That'll give you some operating revenue to avoid it going the way of the dodo.
https://cloud.google.com/appengine/docs/python/#Request_Head...
and it would likely be free if the number of requests was below a certain threshold per month.