Why you should not pirate Google’s geo APIs | Hacker News Reader

Why you should not pirate Google’s geo APIs

petewarden.com

309 pointsspenrose113 comments

FWIW, Google doesn't particularly like it when people use APIs outside the terms, either. Having to take down an app is never fun or enjoyable for anyone involved. And the cases in which the terms aren't programmatically enforced right away isn't necessarily a choice; often it is difficult to differentiate at scale between legitimate usage and disallowed usage.

And while I don't know how every other company writes their developer Terms of Services, I do know that in Google's case we measure each word carefully and try hard to strike the right balance between offering as much as feasibly possible, while trying to create something long-term sustainable (so even if the usage grows far beyond our initial goals, we're still in a position to keep them available). We're hardly perfect at it yet, but we're certainly getting a lot better over time.

Also worth noting, often times these terms are designed to honor upstream data contracts, so if for no other reason, it's worth trying to respect them because it's the right thing to do for the other relatively small parties, too.

I would have some sympathy with this if you made it easy for single people to pay you to be able to use your data in a manner outside of the ToS of the free APIs.

However, for the Elevation API there are two choices:

  * free and heavily restricted
  * business api, which looks extremely sketchy †
    and is blocked to most people

The gaming world has learned this with Steam and friends already. People will pirate the hell out of digital offerings that are not available in a reasonable manner. Steam recognized this and made games available in the most convenient manner humanly possible and are making money hand-over-fist. Sure, there are still pirates in gaming, but those are the people who are either paranoid about Steam or those who truly couldn't afford it anyhow.

You're still in this pre-realization phase and have only one single entity to blame if people "misuse" your APIs: Yourself. Make the APIs available for normal people in a reasonable manner and you will find adherence to your ToS becoming the norm, not the exception.

† By sketchy i mean the facts that there is zero pricing information, it's a buzzword-slaughterhouse, and the only way to get at it is to fill out a form with a lot of information, an indication of what one might want to do and which offers no further information or even actual flesh-and-blood human contact.

I don't understand this feeling of self-entitlement - who are we to judge whether or not an API has been offered "in a reasonable manner?" I'm guessing it's very resource-consuming to collect, process, and maintain the data necessary for geolocation APIs (especially given the upstream providers).

It'd be one thing for Google to be deceptive about it, offer a free service, and try to upsell you every point along the way. It's quite another thing for Google to dedicate an entire section in the API intro page [1] about Usage Limits.

Saying that "Google has nobody to blame but itself" is interesting when the blame is usually from developers to Google, not the other way around.

[1] https://developers.google.com/maps/documentation/elevation/

I agree in part with your point, but I think Google has pulled a bit of a bait and switch here. For many years, they were conspicuously open and friendly, as you would expect from a company run by engineers. That got them a lot of geek love, and a big boost to their early adoption efforts. I also think it had the effect of stunting the growth of alternatives. (E.g., Google Reader.)

Now they are running things more in standard large-business style. There's a natural rebound from that as people adjust. I agree the pragmatic thing is to drop the high expectations; I don't think Google is going back. But I can see why people are going to get all Kubler-Ross on Google while they adjust.

By open and friendly you must be talking about small pet projects released by single or small groups of engineers that didn't take much effort to produce.

If you are expecting unlimited free access to some thing like maps which employs many people and probably millions of dollars to produce, without expecting any type of return to sustain further growth...well it's time to rethink your strategy if you plan on staying in business.

Why does HN believe companies that spend millions of dollars to provide services, many times for free, and for pay with restriction are somehow "evil" (since we're talking about google this feels like an appropriate time to use this irrelevant phrase again).

You've missed the point that Google has always been an advertisement company.

I'm aware of that. But they also came out of an academic background, used a lot of open source, published many papers, and built an engineering-focused culture, not a business-focused one.

That is, sadly, changing. But it's not a necessary change.

I would offer that any potential or actual user of the API is an apt judge of whether the API has been offered "in a reasonable manner."

As it currently stands, one has the option of 2,500 requests per day for free or a minimum of $10,000 [1] and a limit of 100,000 requests per day.

Is it unreasonable to wish for at least one option between the two current choices of $0 and >=$10,000? The parent isn't being self-entitled, he or she clearly stated, " ... if you made it easy for single people to pay you to be able to use your data ... [emphasis added]"

[1] "What is the cost of Google Maps API for Business?" - http://www.google.com/enterprise/earthmaps/maps-faq.html

So why was it outside the ToC to take the five snapshots for the Crime Doesn't Climb gif?

I do not know if the Crime Doesn't Climb usage was against the ToS.

They could have possibly made all their requests within the rate limit imposed by Google. I'm unsure, however, if their usage of the data was allowed: "The Elevation API may only be used in conjunction with displaying results on a Google map; using elevation data without displaying a map for which elevation data was requested is prohibited." [1]

That said, I certainly think it's reasonable to want Google to allow paid access to their API at a rate between their current free offering and their prohibitively expensive offering for personal and small business usage.

[1] https://developers.google.com/maps/documentation/elevation/#...

I actually talked to Google yesterday. Minimum tier is 100k per day at $17.5k per year. Very fair price (If you think that is expensive, then you clearly do not work in commercial GIS, where licensing of software and data is insanely expensive). At the same time, they force you to use the data on a Google Map component (yes, even for the paid accounts), and while their api is acceptable, their js map control is severely lacking in features compared to something like leaflet, openlayers, or esri's js component. This may have very well been simply an arbitrary decision, but it has the effect of snuffing out any small companies or start ups that want to use their apis to build new, innovative applications. Considering the big few companies in this space bought up just about all the data companies that originally compiled the underlying data, this seems completely anti-competitive to me. I am not saying they should give it away for free, but there should definitely be some smaller paid plans with more open terms of use, considering that Google and the couple other competitors basically just purchased a cartel together over the last decade.

    their js map control is severely lacking in features compared to something like leaflet, openlayers, or esri's js component.

I work on the Google Maps API. I'd love to hear more about what features you're missing.

Honestly, I had to take a look at in depth to respond, since it has been a while since I looked at the google api in depth. What I still notice is that Openlayers and the esri allow for a lot more service types. You have KML and GeoRSS, which are fine, but do not seem to support many other of the more common formats out of the box such as Arcserver services,WMS, or WFS (which are probably the 3 most common types in the GIS world). I have seen some workarounds for these, but they seem hacky and slow in the implementations I have seen. I do not have a big problem with KML, but I cannot always change what format a 3rd party serves a map as.

The other thing that appears to be lacking is front end geoprocessing. With esri at least, you can do quite a bit of geometry manipulation out of the box. This can be extremely useful for realtime data visualizations, as starting up a new map service is often a bottleneck with any of the backend platforms. If you can just render the new layer on the client, you save loads of time.

My last issue might just be a lack of understanding, but from what I have seen of the backend offering that google maps enterprise offers, it is missing some major features. For one, the documentation is lacking, and it appears that the entire rest api is marked as experimental. Second, although it allows for some basic hosting and whatnot (easily replicated by a free server like geoserver, which is trivial to set up), it once again does not have any real processing engine, so you have to download all of the data to another server you own anyway to do any real work on the data.

That being said, I really do think it is a great product, and I definitely point people in your direction if they are looking for a quick an easy way to add a map to a site with some basic data. At the same time, I am not sure the enterprise plan really makes much sense, since the results can be so easily replicated with geoserver + openlayers and you already need a server solution anyway for running a processing engine like grass or pyqgis.

Also note: I am probably not your typical use case, and I realize that most people just need to throw preprocessed layers on a static map. All I can say is that I look forward to the day when I do not feel like I am the only one around doing dynamic big data GIS on the web, so I don't have to keep building all of this stuff from scratch ;)

I also want to mention that I retract the part about leaflet. I think you guys have wayyy more features than them, but they are the new kids on the block so we can cut them some slack. Also, their controls are the best designed for aesthetics and usability IMHO, which is no small matter.

It's not really entitlement. It's part of market forces. People say good things about products that are sold well, and bad things about products that are sold badly. Negative terms create negative buzz and drive people away. This is how it's supposed to work. And 'negative terms' is up to the market to decide. Obviously some people are unreasonable, but they're usually outliers.

Building your own elevation API is absolutely trivial; I don't know why anyone would even want to pirate Google's API for large-scale usage.

Firstly, the data is freely available. It's NASA's SRTM [1], downloadable from a zillion mirror sites.

Then just take some code to calculate a lat/long offset and find the right position in the right tile. Bob Osola has some easily portable PHP for this if you need it [2].

Free hint: Putting both your tiles and your (Lua) lookup code in Redis, then calling the latter with EVALSHA, is a really neat and fast way to do this.

If you really need ocean depths, higher resolutions than SRTM offers, or stuff above 60 degrees latitude, then it gets a little more complex. But for 90% of cases doing it yourself is eminently practicable.

[1] http://www2.jpl.nasa.gov/srtm/ [2] http://www.osola.org.uk/elevations/

interesting. I think of these data are freely available, then there ought to be more people using it and making competing versions so that google will not have a monopoly on provision of such data apis.

Don't wait for "other people;" if you think it should be done, I'd encourage you to do it yourself!

Building a robust elevation api is definitely not trivial for the average dev using the google api. If someone thought the only way to get elevations was through google then they definitely will not know how to find and compile the data, then write the TIN (triangulated irregular network) algos necessary to appropriately estimate the elevation of an arbitrary point. No, it is not that hard, since tools like Grass can do a lot of the TINing stuff for you, but just running Grass, much less scripting it, is not trivial for someone unfamiliar with GIS. Making this stuff stable and performant on a server (as opposed to a one off calculation on a dataset) is also not trivial in the least.

Worrying about TIN is the 10% case. Simple interpolation within SRTM raster tiles is all that's needed for 90% of cases (IMHO, obviously).

I would not think interpolation would be that much easier than a creating a TIN. I have not looked at the dataset though, so it might be "pre-interpolated" (probably using a TIN if I had to guess), in which case this would be pretty easy with something like pyqgis on a flask server or whatever. The entire api could basically just be:

    @app.route("/elevation")
    def elevation():
      fileName = "/path/to/raster/srtmRaster.tif"
      fileInfo = QFileInfo(fileName)
      baseName = fileInfo.baseName()
      srtmRaster = QgsRasterLayer(fileName, baseName)
      res, ident = srtmRaster.identify(QgsPoint(request.form['lat'], request.form['lon']))
      return ident[0]

I can test this out and throw it on github with some polish if anyone is interested.

How do you know it is just SRTM?

Sorry, should have made myself clearer. I'm not saying that Google just uses SRTM - I'm sure they throw lots of other sources and general Google Magick in there. But for most people in most places, SRTM is good enough.

np and cheers!

Yeah, I think so too. And querying SRTM tiles is prerry simple.

There are many problems with the ownership of data, entitlement, etc.

I've argued with the founder of delicious endlessly about the API limits they imposed which ultimately meant that nothing interesting could be done on the outside with delicious. Nothing interesting ever happened on the inside and ultimately it got sold to Yahoo! and destroyed. I guess the founder got some cash, but the data that was contributed by the end users was destroyed.

Nobody asked them for permission to sell to a psychotic company, have the site destroyed, etc.

Towards the end spammers found that they could (within the API terms) get endless amounts of legitimate 'cover traffic' to cover their links.

In the specific example of the elevation data set, the data for the United States looks like it's public domain (USGS DEMs). The rest of the world may be more difficult to get high resolution but there is STRM which is reasonable ok for a lot of uses.

About three years ago I got in contact with some very nice people at NASA and/or the USGS (I don't recall which agency ended up being the end-point) for what, at the time, was some of the best data available from the joint ASTER missions they ran with Japan.

The process was essentially to fill out a form and send them a new, unopened hard drive. What I got back a few weeks later was about 125GB of high resolution GeoTIFFs. Along with it were PDF's explaining how the data was rectified and cleaned up as well as information for how they encoded the data.

The process and interactions really impressed me with the quality of people involved in agencies like the USGS and NASA.. just like the people who provide API's in the tech industry, they're usually really in to what they do and interested in hearing how other people are making use.

The data is totally out there and you can get your hands on it without a) breaking terms of service or b) needing any sense of entitlement or self-righteous "ethics". I think the article hit the nail on the head. In that ignoring the data freely available reduces its viability. Projects like GeoNames, OpenStreetMap, the deactivated OpenAerialMap are all incredible sources of information that we can't afford to ignore in favor of the easy way out

I’ve worked with NASA and USGS a fair deal both for fun and professionally, and this is typical.

They are extremely competent and sincerely want you to have good data. They are also hampered by the bureaucratic limits of any large organization. So it’s like working with a large, well-run business that’s hired a lot of the best people in its field and is working on good problems, yet is large enough that it can’t move to meet the exact needs of any one customer.

But on top of that there are political concerns. They have an institutional fear that a congressperson in a budget debate is going to stand up and say something like “And apparently we’re paying the Geological Survey $N million a year to run a web server for something called geotiffs that tell you how tall hills are!” That’s my impression from reading between the lines, anyway; no government employee I know has been indiscreet enough to deliberately hint at such a fear.

For example, the best interface to SRTM isn’t from the agencies that made it, it’s a single-page project from Derek Watkins at the NYT: http://dwtkns.com/srtm/

Working with NASA in particular feels like working with an industry leader that has a mysterious policy against advertising, or even going out of its way to help you find resources. (Individuals do, but not the organization, at least not anywhere near in proportion to the number and value of its resources.)

NOAA too: they have some amazing satellite imagery that’s public domain, but they simply do not have the budget to do anything but the most halfassed job of hosting, publicizing, and documenting it, because from a funding perspective that’s frivolous. They barely archive their images, because no one with budget control gets why a weather agency should save its input data. Look up “VIIRS granule” – that’s technically open data, but yikes.

The resources are there, and if you make the effort to figure it out, the people who manage them are pretty much all a delight to work with. But you have to deal with the damage created by a political culture that too often treats our civilian space and geospatial agencies as afterthoughts rather than as highly multiplied public goods.

With regards to your political argument, its something I've often thought about because it can go either way. There's the loaded language approach as you mention, but there's also the possibility of the hypothetical member of the House or Senate saying "And apparently we’re paying the Geological Survey $N million a year to run a web server that no one is making use of and funding data that no one seems to care about."

This would be a hard argument to defend against as it directly hits the value derived from the service rather than the idea that the service itself exists. In either case, I couldn't agree with you more that the civilian agencies really get a raw deal (and I'm sure its only compounded by the mostly stagnant level of people graduating with STEM degrees[1])

[1] http://www.nsf.gov/statistics/digest12/stem.cfm#3

Really, don't bother trying to contact them, it's not worth it. If anything their business API is less sketchy than their actual sales tactics.

We had two different conversations with them and the first time the rep quoted us a rate that worked out to $72 per 1000 map impressions. The second time we spoke to a different rep who, when asked the differences between the free plan and the business plan, told us that if we were currently on the free plan outside of the ToS there would be "consequences".

Edit: I should add that afterwards we immediately switched to MapBox. Straightforward pricing, and it feels great to know that we're supporting the open source community.

uh, even at its highest it was $4 per 1000 (and now it's 50 cents) and you get 25,000 free per day. And that's through the automated API console, no enterprise license. And the "consequences" are...you get an over quota message. Seriously, search for "OVER_QUERY_LIMIT" and then compare that to all the stories of people being sued and shut down because they went over their maps API quota. I'll wait.

I've heard of terrible sales calls with them, so maybe that's what happened, but this seems a bit of a stretch. If it did, I hope you did some basic research before deciding what to go with.

In any case, MapBox is great and they deserve the support, so at least it worked out.

Yeah, we had done some research before going into the call, which is why we were just floored at what they were saying. "So you're saying that an enterprise license is more expensive than the business rate listed on your website?" ... "yeah."

I think the whole google maps enterprise was just somebody high up saying "hey, maps is popular, let's try monetizing it" and then not actually holding any of their sales people accountable for anything.

"I would have some sympathy with this if you made it easy for single people to pay you to be able to use your data in a manner outside of the ToS of the free APIs. "

I'm not sure why you assume, among other things, that upstream contracts always make this possible.

Occam's Razor makes me think that Google simply does not want to "bother" with many low-value customers, associated support infrastructure, etc.

Honest curiosity though: What kind of contract would make it impossible and would one make such a contract?

An upstream contract that required detailed information about paid users, for instance? I could imagine wanting information like that for auditing if there were complex restrictions on permissible resale of the upstream data.

"Honest curiosity though: What kind of contract would make it impossible and would one make such a contract? "

Most of them :) They often have requirements on downstream users, such as requiring we provide info about them, or that they be restricted to certain use cases, etc

There are a lot of ridiculous restrictions i've seen in my days. Most data brokers don't really want customers to go through you to get "close to bare data". Thus, they give onerous contracts, and restrict how you can sublicense the data, as well as usually requiring attribution, in the hopes that people will see your restrictions, see the attribution, and buy the data from them instead.

I thought that was the original point that was missed too. Don't do it because the provider is telling you not to.

At least in this cited case, it was a throwaway project that won't be around, but folks who copy the code may be in for a shock.

Here's another reason that I wish was more prevalently discussed: it's not okay to violate Terms of Service anyway. Don't accept the terms? Don't use the service. We'd all have a field day if "big brother" violated the terms of an open-source license, but the freedom we enjoy in the community comes with a responsibility to honor our agreements.

I just wish people would stop being so self-entitled period.

Agreed. The tone of the article is correct, in my opinion, because this really does hurt the community. I think we'd have an easier time fighting things like SOPA, DRM, etc. if actual piracy wasn't such a big problem.

And if theft didn't exist, we wouldn't need police.

This is a useless observation. You may be able to effect a limited degree of societal change, but you will never satisfactorily eliminate the problem at its source. Other solutions are necessary.

I feel like I deserve to live in a world with fewer self-entitled people.

As my developer friend likes to say - this is why we can't have nice things.

Violating the terms of a license is very different to violating the Terms of service you "sign" with a button click.

Actually I'd argue that you're more obligated to abide by something you explicitly checked "I have read and accepted..." than something that was listed somewhere in a file you downloaded with some other stuff. You probably just happen to like the terms of the GPL or ASL more than you like the terms of Google's API. I do too - but at what point are we just losing all integrity because we're too lazy to read a long document we have no intention of actually agreeing with?

If there is an actual click-through and the license is understandable by a human, perhaps. Usually there is neither, though.

I rarely even know how to look up terms of an API, or that there is an API. If I know the name of the API, I can look for "Google Foo API", but that name isn't always obvious. If I just see a URL with an obvious schema, and I change the HTTP GET parameters to retrieve different stuff, am I allowed to use that in, say, a small-scale art project, or not? The usual internet default is that public URLs are implicitly giving blanket permission to retrieve them, except that retrieval by spiders should abide by robots.txt. Google itself makes heavy use of this ok-to-retrieve default assumption.

If I go to http://maps.google.com there is no license presented to me as a condition of using the service. I can however look for fine print and seek out the link "Terms of Service" of my own accord, at which point I get this: http://www.google.com/intl/en_ALL/help/terms_maps.html

Those terms specifically forbid several things I'm definitely not doing (mass-downloading data, using it for turn-by-turn navigation, etc.), so my first read of it makes it sound like grabbing Maps content in my art project is okay. But then I'm not a lawyer, so maybe it isn't okay.

Ridiculous.

Playing dumb isn't an excuse. Can I stop paying my morgtage on my house because the 40 pages of documents I signed I don't 100% understand? No.

If you want to enter a legally binding contract that you don't understand you hire someone to help you do so, or face the consequences if you signed something you were not smart enough to be signing.

> if you signed something you were not smart enough to be signing

That's good advice, but I didn't sign anything here.

I have in the past signed contracts with companies agreeing to use services or software in certain ways (generally pre-release stuff under NDA), and in those cases I do indeed read the contract carefully before signing it, sometimes redlining parts of it before they countersign, and I abide by our mutually agreed result. If I don't feel I understand it, I refuse to sign, or ask for clarification.

However, to my knowledge, I have never signed a contract with Google. If they had asked me to do so, I would certainly have refused if I didn't understand the terms. But I do not believe they have even asked me to. All I have done is visit their website, just like they have visited my website.

I have a ToS on my own site, but as far as I can tell, Google just accesses my site however it wants and does not read it (they definitely don't pay me for the premium tier, despite exceeding the stated free access limits).

>However, to my knowledge, I have never signed a contract with Google.

To use their API you will need to agree to their TOS

https://developers.google.com/maps/licensing

I have never visited that page. I have visited this one: http://maps.google.com

That webpage serves map tiles over HTTP, without asking me to agree to a contract first. I take it that I may therefore access the map tiles over HTTP? Obviously I assume I cannot redistribute them, as they're copyrighted. And it is good netiquette to observe robots.txt for bulk/crawl access. But beyond that I don't believe me visiting their page constitutes me "signing" a contract with Google, any more than Google visiting my page constitutes them "signing" a contract with me.

Perhaps there is some other API with nice JSON stuff and OAuth or whatever, but that's not what I'm talking about. I'm just doing HTTP GET requests for map tile URLs.

I think you're confusing accessing a public version of their data with accessing their API, which this thread is about. When you use maps.google.com you are NOT agreeing to any terms, because they don't really require any to use their data on their site, but when you use the API, you're required to agree to their terms before being able to use it.

FWIW, there's a "Terms" link at the bottom of maps.google.com which links to the additional ToS for Google Maps/Earth:

https://www.google.com/intl/en_us/help/terms_maps.html

and the first link there is the Google Universal Terms of Service that you also agree to bind yourself to:

http://www.google.com/intl/en/policies/terms/

In what sense did I "agree to bind [myself]" to those terms? I just loaded a public website that has map tiles on it. You are arguing that I have agreed, thereby, to something written on a different website (two clicks away), which I was neither presented, nor asked to agree to, nor signed an agreement to?

It seems very similar in the sense to which Google "agreed" to pay me money if they spider my page more than N times/day, which is what my own website's terms say in the footer. Yet I've never received a check. Is Google playing dumb and claiming they never read the terms, so can't be held liable?

Aren't the same exact APIs are serving as backend to that "public version"?

If so, looking at whatever data's on your wire (browsers are subclass of user-agents for a reason), and then doing similar requests by example does not make you sign some agreement with Google, right?

Just because their app consumes their API doesn't mean that the app and it's endusers are subject to the same terms as the API.

> Aren't the same exact APIs are serving as backend to that "public version"?

Nope.

Maybe if google (and all other companies with similar "walled" gardens) had a leg to stand on when it comes to data, one might be more inclined to care more to abide by their wishes…

Playing dumb isn't an excuse; But if one were to juxtapose the consequences of ignoring google's wishes and not paying ones mortgage, that would seem ridiculous.

And is accessing a url is a legally binding contract now, especially when nothing is signed or acknowledged by one or both parties when the url is accessed?

>And is accessing a url is a legally binding contract now, especially when nothing is signed or acknowledged by one or both parties when the url is accessed?

This is in no way comparable to using a companies API.

Really? Found this[0] link on a site listed on this topic thread.

And like blog posts written by people on various subjects, one might click the links, find something of value from accessing the link and decide to access it in an automated fashion without caring (Oh, it has api in the url! * doesn't think twice about it *) to bother to look for anything else because as far as they are concerned, they're getting what they need right there. Now if they run into rate limits and get blocked for x amount of time, its tough luck. But if one doesn't run into rate limits, and continue to uses such a link, should they expect the 2 am wake up call or expect to get kicked out of their home or face significant hardships in life because of it?

[0] http://maps.googleapis.com/maps/api/geocode/json?sensor=fals...

They should expect to get cut off from accessing Googles API at any time if they are not operating within the bounds of whatever Google has deemed appropriate usage.

I agree with that statement, and I mentioned that possible scenario above which ended with that being "touch luck" on the person who decided to access the url in such fashion. But nowhere in those potential interactions (that do happen all the time and not just with google) in the above, magically create some legally binding contract between parties.

Now If it is 'malicious' enough for google to recognize the access, and trace down the source to get the DOJ to throw the book at the person/company, that is a different story that we have seen before…

Actually, given the events of the last few years, it's entirely possible to stop paying the mortgage on your house by pleading ignorance to the documents you signed.

More to the point -- regardless of the specifics of the law, the persistent desire of this forum to correlate legality to moral correctness is somewhat disturbing. What does it even mean to form an agreement with "Google"? Even assuming the historical basis of contract law, what constitutes "consideration" in an agreement like this?

Personally... I don't see any ethical reason to perform negotiations with organizations that have become as large and as disconnected as Google. I speak to employee after employee and they all speak of Google's commitment to privacy, to openness, to not doing evil; but the actions of Google at every turn display otherwise. A man can make an agreement with another man or perhaps even with an organization that is capable of coherent action, but Google is neither. No "agreement" with Google is valid if only because Google is not a collective that one can actually negotiate with, never mind that it's proven time and time again willing to discard agreements and promises in the name of profit.

Finally, the statement is often made that if one chooses not to consider Google as an entity with which one can negotiate, one should simply not utilize their resources. Why? Because we can't negotiate with a mountain, should we fail to mine the minerals therein? Google is a mound of computational power and data with no remaining hold on human morality or ethics -- anything we can retrieve from there is salvage. Take it and use for the betterment of humanity, because they certainly will not be doing so.

> my first read of it makes it sound like grabbing Maps content in my art project is okay.

You only have to read as far as the beginning of the fourth paragraph to find: "Unless you have received prior written authorization [...] you must not: (a) copy, translate, modify, or make derivative works of the Content or any part thereof..." etc.

I'm not sure if you're being intentionally obtuse here, but maps.google.com is the public Google Maps website, while the Google Maps Javascript API lives at https://developers.google.com/maps/documentation/javascript/. The very first page there says:

The JavaScript Maps API V3 is a free service, available for any web site that is free to consumers. Please see the terms of use for more information. ...with a link to the full TOS for the API: https://developers.google.com/maps/terms

You're not at all obligated to abide by an open source license that just happened to be bundled in with a bunch of files you downloaded.

The thing is, without that license, you're bound by copyright. That means you're not allowed to redistribute those files, or anything derived from them, at all without permission from the copyright holder, with the exception of uses allowed by fair use.

For source code and similar things, there's a default position which is highly restrictive, and the open source license loosens the restrictions. Nothing forces you to accept the license. It's just that you can do much less with the material if you don't.

Terms of Service are a completely different beast, because the default position is that you can use services that are mad public over the internet more or less however you feel like, as long as you don't violate copyright and such. The ToS then adds restrictions.

I think the point you make is an excellent one. However, we've seen court decisions that essentially equate ToS violations with wire fraud, no? I wonder where the line gets drawn...

> we've seen court decisions that essentially equate ToS violations with wire fraud, no?

I can only think of one, and it was overturned. Most of the others (that I can think of) have to do with an employee/employer relationship. Even the recent Craigslist case wasn't necessarily a case of ToS violation, because they were explicitly asked to C&D, but didn't.

And we've also seen antimonopoly actions against companies abusing their power position.

My point is simply that the two are not at all equivalent. An open-source license grants additional rights to the person receiving the license, while terms of service restrict the person receiving it. As such, there is no inconsistency whatsoever with saying that one should not be automagically bound by such things just because they clicked through them or they happened to be sitting on a web site that you use. It doesn't matter at all if you violate the terms of an open source license. All that means is that you don't enjoy the additional rights granted to you by that license. You can still enjoy all of the rights granted to you by normal copyright, which means that you can use your copy and make additional copies in ways covered by fair use. If you go beyond that, you're violating copyright, and that is the fundamental problem, not violating the license. Violating the license is only a problem when you want to use some of it (e.g. the permission to distribute derived works) but not other parts of it (e.g. the requirement to distribute source code).

Violating a license isn't illegal, only violating copyright is. I don't think there's an equivalent for terms of service for a freely-available service.

I really don't see the difference. Either way you're agreeing to abide by a set of rules by the owner of the service. If you don't like their rules, use one of the alternatives.

OK, so on the one hand there's white (or grey) hat cases, and on the other there's spammers and scammers. Both break the ToS.

I think a big difference is whether it's for commercial purposes, or has some evil motive. Eating KFC in McDonalds is probably breaking some rule, but no-one in their right mind would really care. Selling your own fried chicken in McDonalds is much more likely to lead to trouble. A political stunt is somewhere in between, but should generally be given a bit of lenience.

So I'd say it should be OK to use a script to automate some personal task. Creating a business which relies on breaching a ToS is both wrong and stupid. And while political hacks might be a bad idea, they might also be the right thing to do.

Here's the thing about the law - it's not quite black and white. A judge or jury will always consider the circumstances, when they decide what to do with you. The same is true about companies blocking you. Sometimes, you need to use common sense, and hope that other people do to.

This really could use more exposure. People who believe that because they only had to click "Accept" they may claim ignorance or stupidity/absurdity of the terms the accepted ought to be corrected. It's simply silly to think you may violate the terms of any agreement made, especially when that agreement is necessary to your use of the service (hence "terms" of service).

While it is clear that hacker culture embraces putting things from numerous sources together, in this case the Google Maps elevation API with crime data, it's one thing to make a cool presentation and another to make a cool presentation derived from someone else's work against their explicit wishes, expressed in the terms.

The use of software implies (by necessity or otherwise) that you wish to make/create/analyze/do-X with the software, and by using it you've accepted the fact that you will do-X, but within the terms.

It's a funny world we live in ;)

If they wish people not use it in that way, then they should make it impossible to use in that way, or ban those people from using it.

Those are really the only two options, no matter how much we wish people would be honorable.

Personally, I think it's silly to think that you may be bound to the terms of an agreement just because you clicked a button that says "Agree", without any of the fundamental attributes of a contract being present, and without even interacting with a representative of the other party in any way.

I've noticed typically I agree with your views, but on this case we diverge.

I think it's reasonable to assume that when the service cannot be accessed before the accessor clicks "agree", and the accessor clicks "agree", that constitutes a valid contractual agreement. It's no more silly than mailing in a credit card application and being bound by those terms.

The fundamental attributes are present. There is a contract and you can agree to it or not. Your interaction with the company is viewing the terms the lawyers wrote and agreeing to them or leaving the website.

Do you also agree that a person infected with malware also accepted it just because he clicked "Ok" or "Yes" on some of the dialogs?

There is no signature, there is no proof of identity of the signing party (copy of personal ID, witness, ...). Usually no confirmation that other side also accepted it. No copy of ToS received after "accepting". We all know that it could be that, for example, my 4yo child clicked on it when I went to the toilet after reading first few paragraphs of ToS, and now I'm somehow bound to it? Also, it happens very frequently that you "accept" some ToS, and after two years you find on some forum that Google, or PayPal just changed their ToS without sending you the updated copy for acceptance.

Sorry, but too many things that are really different between ToS and a real acceptance of a Contract.

If you agreed to a ToS by clicking yes, and the ToS allows them to install malware that's a failure on your part to read it. If your child clicks it, you probably have recourse in civil court because a 4-year-old is not able to execute contracts. Most ToS's include a clause that they can change it at any time; you might not be legally bound by the new one, check that with a lawyer.

It seems like you are angered that you have to enter into a contract before using software, and you would prefer to just click "accept" and have that magically not be interpreted as you agreeing to the terms. "I'll use it but I don't accept the terms". I'd like that too, but I'm afraid that's not how it works.

No, I just say that the fact that someone clicked accept button on some website is not a proof that I accepted anything. If things worked like that you would not be required to go to the bank to sign a contract and take a loan, one mouse click on bank website would be enough.

I could as well write whatever ToS on my homepage with "accept" and "not accept" link with whatever terms, and then wait for Google bot to "accept" my terms. Do you think Google would be bound by their "acceptance" in the same way.

It's not like I was using my digital signature or whatever, it's just that someone clicked somewhere.

The burden of proof here is on the "vendor" or software provider. They need to prove that I accepted their terms, not the other way round. That's why many free services require credit card, copy of the ID and physical signature as a proof of identity. The fact that Google decided that this might be too expensive for them to enforce, and that is cheaper to just not do it is their choice.

Well I think the burden of proof is going to be on you if you are found willfully using the service afterwards. Otherwise you're probably right.

I'm not a lawyer, so take the following with much salt. Furthermore, I'm arguing more from an "ideal legal world" viewpoint rather than the current American legal environment.

A contract must consist of:

1. A "meeting of the minds." This means both sides come to a common understanding of what's in the contract. Basically, it means there's an offer by one side and acceptance by the other without changes.

2. Intention to be legally bound. Basically, this means that the context of a contract matters. It seems to boil down to commercial agreements generally being enforceable while others (e.g. promising money to family) are not.

3. Consideration. This is some sort of exchange of value. In short, a contract cannot be one-way. This is why you sometimes see e.g. things being sold for $1 rather than being just plain given away for free.

The way I see it, there are serious problems for click-through agreements for both 1 and 3. 2 is not a big deal, as despite the name, it doesn't appear to actually apply to the case where you click Agree without the intention to follow the contract.

One problem with #1 is that essentially nobody ever reads these things. In general, "I didn't read the contract I signed" is not an excuse, of course. However, I think this does (or should) change when everybody doesn't read it, because it means that the other party knows, or should know with reasonable certainty, that you're unaware of the contents. Normally you presume that each party is aware of the contents of the contract, but you can't do that with click-through agreements since it's widely-known that they go unread. A contract which I don't read and which you are well aware I didn't read should not be enforceable. For high-value, important contracts like buying a house, a person familiar with the contract will go over each page with you and have you initial it to affirm that you read it, precisely to avoid this problem. The combination of widespread failure to read the contracts and a complete lack of effort to ensure that you, personally read the one in question means that, IMO, there is no meeting of the minds.

Another problem with #1 is that there is either no opportunity to propose changes, or the other side never notices or reviews them. With a normal contract, I can cross out clauses, add or change wording, etc., at which point the other party can accept or reject my changes, or propose further changes. Where this really runs into trouble, I think, is when you edit the contract locally before clicking Agree. This is generally trivial using a DOM inspector for any click-through agreement found on the web. Imagine I take the ToS in question here and alter it to read, "In exchange for mikeash's use of the Google API, Google promises to pay mikeash the sum of ten million dollars." Then I click Agree. Should Google be bound by this agreement? I imagine you immediately and strongly say "No!" But why should I be bound by the original while they are not bound by the alteration? In a normal contractual setting, they have the opportunity to say yes or no to my proposed changes. They will, of course, say no. In this setting, they don't say no! They simply grant me access to their service after I "Agree" to the altered contract. Does this not imply acceptance on their part? If not, what's the difference between their situation and mine?

Finally, #3 seems to completely destroy the concept of any click-through agreement that isn't part of some sort of payment process. Terms of service presented as part of a checkout process would seem to be fine in this respect, but when presented as part of a free web site, it doesn't seem to work. They give you access in exchange for... nothing. Similarly, click-through software licenses seem to fail here because they show up after the money has been exchanged. I pay for a copy of software which I obtain, and only later am I presented with the EULA. Since the money and product already changed hands, there's no further consideration, and so the EULA should not have force.

That's my thinking. I welcome dissenting opinions.

They probably are fully aware that nobody reads the clickwrap, that's an interesting point. It would be interesting to hear what a lawyer says about that. I don't think it's a truly reasonable defense and I wouldn't want to rely on it. (If we're thinking "ideal legal world", then I'd like clickwraps to be human-readable and people to actually read them.)

But I think there's a very clear distinction between a simple agreement with the original and with altering the DOM. If you are clever enough to alter the DOM you definitely know that the alterations are not going to be sent back to the server. You also know that agreeing will let you use the service because the server will think you agreed to the original. That seems like a clear-cut case of fraud to me. I don't see why an agreement to the original contract is invalidated because you weren't offered the opportunity to make a counter-offer (I agree that's a shitty situation, though). That's why I think it's like a contract in the mail; you get an offer and you can agree to that or not. You can make a counter-offer but it will be binned.

Consideration isn't solely concerned with monetary purchases, is it? I imagine they can construe your incoming data as valuable. It would be interesting to know if sites like Reddit can claim user interaction has inherent value for the site and for the users. I guess the EULA would be granting you license to use the software, but would not give any additional value to the developer so consideration does seem to invalidate that.

You're probably right that a ToS is going to be hard to uphold in civil court.

Anyway, thank you for elaborating, I'm glad to get a better understanding of your perspective and contract law.

There was a story floating around recently about a Russian man who marked up his credit card agreement and sent it back, where it was blindly signed by a representative of the card company, and he then enforced it on them with hilarious and expensive consequences for the card company.

Yeah, they'll probably just toss your counter-offer. But the point is that you can make it. You can take their contract, alter it, sign the altered copy, and send it back, and then they can accept or refuse. The important point here is that you are not considered to have agreed to the original because you signed an altered copy.

A more thorough way to do this electronically would be to modify the DOM, save the modified agreement, click Agree to sign your end, and then e-mail the modified agreement back to the originator and let them know what's going on. Of course, their automated systems have given you access in the meantime, but that's hardly your fault.

I don't really understand how making a modification you know won't be sent back to the server is "fraud", but sending the user an agreement you know won't be read is "not fraud". Seems like either we base both scenarios on what people should do (read the original/modified agreement), or on what people actually do (not read anything), but not base one scenario on one and one scenario on the other.

You say you don't see why an agreement to the original is invalidated because you couldn't make a counter-offer, but that's how contracts work. Negotiation is inherent to the process. Big companies are trying to hide that with form contracts that everyone is expected to sign as-is, but it's still supposed to be present.

I've said the same in the past and usually it's to the business managers or executives. Too often they are willing to conveniently overlook licensing or service restrictions in order to get what they want or need.

A specific example - I had to convince executives that we needed to pay MPEG LA because we were using FFMPEG to encode and decode using 264 on our servers. I only convinced them after a long debate and contacting our legal department to explain the risk. It had very little to do with my personal views on licensing or patents and everything to do with professionalism. A lot of the other software developers thought this was wasting money and time, and paying off people that were somehow evil because they charge large sums for using their patented software. My stance has always been if yo don't like the terms use something else and avoid the ethical dilemma. It's just plain easier and avoids some risk.

I've been meaning to put together a more technical blog post about the various geo pieces that we've released at foursquare. But for now, here's one that's an overview of the things we've been working on: http://engineering.foursquare.com/2013/06/11/quattroshapes-a...

The biggest piece is http://twofishes.net/, our coarse (city/neighborhood level) geocoder based on geonames data. It also serves as our coarse reverse geocoder when we combine it with polygon data, which we've released at http://quattroshapes.com/, as a combination of open government data and synthesized data.

http://zetashapes.com/ is a side project of ours, where we're seeing if people want to draw their own crowdsourced neighborhood boundaries in US by coloring in a map of basically city blocks.

I'm always eager to chat about open geodata as well as complimentary commercial solutions.

His site http://www.datasciencetoolkit.org/ certainly looks interesting! One of the most impressive kinds of arguments - "I think you should stop doing x and here are the resources y I've created/curated to make it easier for you to do so."

I was thinking the same thing. Lots of people get the first part. But not so much with the second part.

From the article: "I’m not RMS, I love open-source for very pragmatic reasons. One of the biggest is that I hate hitting black boxes when I’m debugging!"

RMS started the free software movement because he wasn't able to adapt a black box printer.

Even if RMS started from a pragmatic standpoint, his position has clearly moved far beyond pragmatism.

Has it? I think it's very pragmatic to want control over your hardware and the ability to repair problems or add functionality as desired. If you consider it from what is best for society as well, it's very restrictive to have hardware where the software cannot be modified.

I wonder how much of this is an outgrowth of 'hackathon culture' which is to say "Hey we can do this thing ..." and get to some demo without having the foundation bits worked out. (or more importantly the future foundation bits).

There are a lot of people who used to make 'kit cars' which was they took a VW bug chassis, removed the bodywork, and installed some fiberglass body on it. Really "cool" looking sports cars. But you couldn't really say "We're going to be the next Porsche selling our new "Wangdingo" sports car which, as it turns out, is a fiberglass body on top of a VW bug chassis. The difference between kids making a prototype which looks cool (professional even, some of these cars were really really nice) but as a 'new car' it was always a cheat since the other bits weren't engineered, they were 'acquired' and outside the plans of the original engineers for the chassis. Compared to the 'motorhome' business where the company buys a Chassis from Ford or Chevy as a chassis to build a new vehicle on.

Makes for an interesting conundrum. You can fail fast and iterate prototyping like this, and you can arrive at a saleable product for which there is no route to production because the underlying tech you rely on cannot be used 'legally' the way you want to use it, and the amount of money the product might conceivably generate is less than the cost of re-creating the crucial bit of technology from scratch.

It even gets more complicated as `dewitt mentioned: "And the cases in which the terms aren't programmatically enforced right away isn't necessarily a choice; often it is difficult to differentiate at scale between legitimate usage and disallowed usage."

This sets up a situation where people/companies can effectively 'get away' with it. So what is an aspiring entrepreneur to do when they are able to take advantage of a situation like this in a way that the other company isn't in a position block it (without throwing the baby out with the bath water) nor are they even trying to do what you are doing when you combine such usage with other "legal" means or data one collects themselves that is providing some value to people they tested their service/'hack' with?

If you're looking for elevation data, the USGS provides this for free for the US.

Example: http://gisdata.usgs.gov/xmlwebservices2/elevation_service.as...

Also worth noting, Adam Schneider's GPS Visualiser website is excellent for quick elevation lookups globally.

Handy for me as Geoscience Australia (another good resource) unfortunately charge for their DEM data.

http://www.gpsvisualizer.com/elevation http://www.ga.gov.au/

Here's my problem:

1. The TOS aren't heavily enforced; thus 2. No one I work for cares about complying with the TOS; thus 3. No one is particularly supportive of my spending additional time to work within the TOS.

Using and contributing work time or resources to an open source project is going to be a hard sell until there is some pressure on the business end to change. Having our data sources cut off, or having to reimplement on different providers would help.

[Edit to add] Yes, you're right. When I compare their map side-by-side the same area in maps.google.com they are clearly different. I should have looked more carefully before asking my question.

So I guess my question should be "If they had the exact same page except using a Google map, would they be in compliance?"

It's not clear to me in what way the "Crime doesn't climb" page violates the quoted terms of use. Would someone please explain it to me?

  The Elevation API may only be used in conjunction with
  displaying results on a Google map; using elevation data
  without displaying a map for which elevation data was
  requested is prohibited.

They used the API to display results on a Google map.

Their site used the elevation data to display a map for which elevation data was requested.

I'm not being facetious. It seems to me that their use matches the requirements. So what exactly is the problem?

The map in question looks like it's actually from Yahoo! Maps: http://maps.yahoo.com/

I don't think so, it says Nokia in the bottom right, along with CartoDB: http://cartodb.com

Indeed, it looks like Yahoo! uses Nokia "here" maps as a data source.

CartoDB is not a map provider, it's a web-based GIS suite that lets you import different base layers.

It's rate limited for a reason and every other so called "better alternative" will have some sort of limitations in place, if they want to stay in business. Nothing is for free. By saying people will just magically make a free, unlimited API service as long as people stop using Google's is dumb.

Everyone should try OpenStreetMap based APIs, e.g. MapQuest Open.

I tried one part of the API recently - I'm building an app that's partially intended to give local cycling directions.

Now I'm between a rock and a hard place. One the one hand some parts of the MapQuest Open API are extremely bad, on the other hand Google has an all-or-nothing license for their geo data, and their native iOS maps component sucks.

MapQuest's cycling directions in Manhattan is worse than having no directions at all. They have a specific cycling flag you can set, but results will blatantly ignore bike lanes and send you down extremely wide avenues with no bike lanes and heavy traffic. IMO using MapQuest's cycling directions, at least in NYC, is an extreme hazard, and I can't in good conscience integrate it into my app.

How do the directions compare to YourNavigation?

http://www.yournavigation.org/

Despite the clunky interface, I was pleasantly surprised by its biking directions: not as good as Google, but not terrible either. (I was in NYC when I tried using it for awhile.) It's fully open-source.

Just plugged in some of the test endpoints I used with MapQuest. YourNavigation seems to have two bicycling modes, "bicycle" and "bicycle (routes").

The former seems to disregard bike lanes entirely and generates some pretty hazardous recommendations (including cycling down 5th Ave, or busy sections of Park Ave in Midtown).

The "bicycle (routes)" mode does better and seems to respect bike lanes, but not consistently. There are some endpoints where it still maneuvers you into dense traffic with no bike infrastructure, when a bike-laned street is nearby and takes only marginally longer. I just checked the OSM's dataset and the data is good - so it isn't a matter of someone inserting a bike lane where one doesn't actually exist.

The app I'm working on is for NYC's new bikeshare system... Still not an acceptable alternative unfortunately. I may just give up and use Google Maps in the iOS app, but the performance of Google's native map component is extremely, extremely bad.

CycleStreets (UK cycle-based journey planner) are on the verge of open-sourcing their code - as in, in the next day or two. It's a highly fine-tuned cycle planner based on OSM data; here in the UK, at least, the results are streets ahead of Google, which will merrily send you down busy, fast trunk roads with no cycle lane.

You could either take their code and run an instance for NYC, or smile sweetly at them and ask if they might host one themselves. Github link: https://github.com/cyclestreets

Yeah, I was doing some tinkering with OSM and found a lot of that kind of additional data (bike trails and such) to be missing. Obviously I should just roll up my sleeves and contribute to OSM, but it still means that I wouldn't build an app or do large research that depends on those kinds of details with OSM as the back-end, knowing how much stuff is missing.

And yes, more municipalities need to start releasing open data.

For what it's worth, they're currently very poor quality to Google's geocoder.

MapQuest Open is very hit or miss. In Australia, the directions & geocoding are pretty good. However, in India it's atrocious - but still good enough for what we need.

just to clear up a misconception, the SRTM is just unusable in it's normal form, you need a lot of time to clean it up and cross it with other sources. This guy does a lot of curation: http://www.viewfinderpanoramas.org/dem3.html

(and I also suck google's elevation data for a test because that's the simplest way to do it: http://nraynaud.github.io/webgcode/test_threejs.html )

Okay, so I have a question: how can open source geocoding services sustain themselves? Doesn't it cost money to gather all of that data? Where does that money come from if the service is free? I would expect an OSS geocoding service to be a) seeded with data that is out of copyright (basic geography and placenames, b) spruced up with any well-regarded modern free datasources (probably government funded), c) spruced up even more from volunteers donating GPS data.

Would a geocoding service be a good candidate for a state-run utility?

If you are looking for a good mapping alternate checkout Leaflet.js. You can load from a variety of tile servers. I think a while back someone had a tutorial an setting up your own tile server on AWS. Actually http://tiledrawer.com/ provides a base AMI. Another good commercial solution for geocoding is http://smartystreets.com/ if you need to do batch processing.

Does anyone know of a good guide to DEM (Digital Elevation Model) formats? New Zealand has some great open data[0], and I expect that it would be a fun exercise to create your own elevation API from it but I don't know how to get started

[0] http://koordinates.com/#/layer/1418-nz-80m-digital-elevation...

I have one comment and one question.

Q: Are OSM/Nominatim good for most countries now? When I tried it was very good to good enough for developed countries like US or UK but for others clearly not.

C: I really dont get Google mindset. They sell geoservices. OK. How much money that would bring 1mil, 10mil or 100mil in over 20bil company. What is the logic? Creating barriers? I dont know.

You would have thought San Francisco's crime rate would have been higher with all of these unauthorised API usages... (Yep, the data was pirated by a guy in San Francisco. Ironic.)

Reply on news.ycombinator.com