OpenStreetMap is the Most Important thing in Geo
gisdoctor.com
gisdoctor.com
I think OSM is an accomplishment on the significance level of Wikipedia and I'm glad it exists.
I don't think Google actually dog foods their Android Google Maps API, so it tends to be pretty limited.
Very common operation is looking for pairwise distance matrix. Pricing and limits are totally irrational.
100,000 daily limit is totally impossible for an app with 10 users. 100x100 = 10,000 :D
Given that Google probably has all of the distances cached somewhere on their servers, I see no technical reason why this limit can't easily be increased. My app can have millions of shortest paths requests per day, and I certainly do not have Google's infrastructure.
I get incomparable functionality using OSM.
https://developers.google.com/maps/documentation/javascript/...
I'm also not aware of OSM offering anything comparable out of the box, although there are some commercial providers who offers similar paid services based on OSM data.
Also, if you think "all of the distances" are cached, I invite you to estimate how many places there are in the world and how big a matrix of every distance from every point to every other point would be.
(Disclaimer: Used to work on Geo at Google. Opinions mine only, etc.)
OSRM is a nice project - out of the box. The lead went to some giant - so project is currently a bit stray but there are also some other alternatives on github which work well but might not be as mature as OSRM.
Google caches insane amounts of things. What one might think is insane algorithms, it's probably just Googles multimillion cluster memory.
Google is also lucky that no one needs the whole matrix, I'm sure that distance matrix is insanely sparse and can be stored in not that large amount of memory.
I understand completely that Google offers these APIs completely for free and they can shut them down easily. I'm willing to pay for more but my quadratic needs aren't met with root pricing.
So, to waste sub-linearly I can easily setup OSRM for my needs :D
What system would calculate a shortest path between Sydney and London? There's an incredible amount of sparseness in the matrix.
OSRM builts a hierarchy that allows fast computation of shortest paths for the whole world map. It takes about 4 hours on my server, fills around 40GB of RAM, and then after that I can magically do insane amounts of shortest path queries.
Paper below has an even faster algorithm that has an operation of finding shortest paths equivalent to just several reads in memory (according to experimental results in the paper, it's roughly equal to 5 reads, meaning it's just 5 times slower than that 1E18 table we'd have). It works on a single workstation.
It might not be a pairwise cache but man, this is some advanced stuff and I'm sure Google's engineers wouldn't think of having 1E18 elements matrix.
That's why I'm a bit surprised by the pricing and the limits. I guess network traffic bandwidth costs.
http://research.microsoft.com/apps/pubs/default.aspx?id=1456...
[1]http://people.mpi-inf.mpg.de/~dmatijev/papers/DIMACS06.pdf
The osm datasets actually aren't that big, the entire PBF formatted dataset is <30GB. You can fit that on any system (not in RAM of course but hard disk access is still a million times quicker than using an online service). There are simple scripts to keep the dataset in sync on your server.
Also it's just completely fucking ridiculous to send a request over the internet to perform a calculation.
I encountered the issue myself ages ago where i had a task to reverse geocode for a railways GPS system for every train every minute. Google and other services would have costed millions of dollars every year, used massive amounts of bandwidth and been waaay too slow. I wrote a library - https://github.com/AReallyGoodName/OfflineReverseGeocode in a day that could easily do millions of lookups each minute.
Also not that you cannot really cache routes, nor could you store all (not even Google) or even just distances or ETAs. But that is also not necessary to get fast and relative 'CPU light' responses.
> I get incomparable functionality using OSM.
Have a look at https://graphhopper.com/ with a reasonable pricing model for the whole Directions API and specifically for the Matrix API https://github.com/graphhopper/directions-api/blob/master/FA...
(note: I'm the author of GraphHopper)
OSS is eating the world of geo.
Other editors can then see the note and might resolve it
OsmAnd is an open source project, however the free OsmAnd version in the Play Store has limitations. You can get the full version (OsmAnd+) if you have money and want to support the developer, but OsmAnd~ has all the features of OsmAnd+.
Well, it’s always funny, but with Google’s data not updated since 2005, they’re pretty useless.
It’s like Google only cares about the US.
I entered a location for the company I worked with. The company moved to another location, so I set about to change the address on Google Maps. To verify I was really the owner of the business, they decided to send mail... to the company's old address (which was occupied by a squatter at the point).
End result, now you have two locations for the company, a correct one and an incorrect one.
I also had a dispute when I edited a location on Antarctica (the Uruguayan base)... editors kept reverting the change.
Similarly experience in Philippines in January, Google maps had basically nothing in Palawan but OSM had great coverage.
The really nice thing about OSM is how easy it is to contribute. There is a famous bicycle route that goes through my town, but it is also famous because it is non-contiguous and completely unmarked on any map. The only way to find it is to ride on the roads and follow the signs (some of which are hand written by residents who live in the area). With OSM it was easy to mark the part of the route that I knew about and now I see a lot of touring cyclists coming through. It's incredibly empowering.
I don't know whether there's anything which could be done to help these more corporate charities get on board (for instance to make it easy for them to set up and maintain an official database using OSM tooling, on the basis that the data they gather is publicly available and licensed as open data).
I've come to the conclusion that it's not malicious (or even 'Not Invented Here' syndrome), but ignorance and a complete misunderstanding of the FOSS world. It seems that a lot of the people and groups involved come from a corporate background where they had to build and keep everything in house. Whenever I mention not reinventing the wheel, building on others work, collaborating with other groups, or opening up our the data I'm met with mostly blank stares. (I could go on and on about this disconnect and why I think it's happening, but it's not directly relevant to your comment.)
I think the major issue is that non-developers (and even some developers) have no idea how to work with others. It isn't that they don't want to or are refusing to, they fundamentally just don't know how to. The idea of working with more than the fifteen people that are present in the room is mind-boggling, let alone the idea of working with people all around they world they may never meet. On top of that simple issue you have the same concerns you do with any person outside the Open* communities: security, trust, liability, etc. Have you ever seen a layperson look at a software/data license? It's beyond overwhelming, so they all go back to whats safe, even if it's not the right thing for their goals or the community as a whole.
To begin to remedy this I think we have a lot of work ahead of us...starting with making the ideas and principles of Openness more accessible to those outside our community.
TL;DR: I believe that silos are a symptom of being ignorant and/or overwhelmed by the Open-anything world.
Google offer a tempting deal to charities: free mapping, and the chance to get your content hosted on Google Maps. So, for example, Sustrans' cycle routes are now integrated into the Google's UK mapping data.
This seems to appeal particularly to the higher-ups. The developers/GIS team may be more au fait with OSM, but when the marketing director says "We have the choice of getting our routes publicised for free on Google, or... what's this obscure 'street map' site you mentioned?", it's hard to convince them. (I did try to persuade one such charity to switch to OSM-backed mapping, but the answer was "Google is a corporate partner of our charity - we can't be seen to switch away from them".)
It's a great shame especially because, as you say, the tooling available around OSM data is so much more powerful than the Google Maps API could ever be. But I'm not too worried: it'll come in time. With OSM powering more consumer-level sites by the day, it's getting easier to answer the marketing director's question.
There is a de facto prohibition on the blending and derivation using many geo data sets due to licensing restrictions and regulatory requirements, including many government data sets. Even OSM does not have the equivalent of a BSD-like license. In many cases, blending two sets of geo data and doing derivative analytics creates an unresolvable conflict between the individual licensing terms and the requirements of a specific application. Specific to OSM, many popular geo data sets have regulatory restrictions that make them fundamentally incompatible with OSM for many applications where OSM would otherwise be a great data set.
With commercial data set providers, the license is customizable to be appropriate for the regulatory and other requirements of the application. With OSM, you are stuck with an OSM license, which would violate all kinds legal and practical constraints for many use cases. I can say that for many geo applications I've supported, compliance with the OSM licensing would literally be violating the law. Fortunately, commercial data sets are much cheaper than they used to be.
- OSM may be a minor input to a petabyte scale derived database, a large percentage of which may be modified every day. Many popular derivations would be technically and economically implausible to make publicly available.
- Jurisdictional restrictions on where the derived database can reside because of the type of data that was used to construct it. Making it available publicly without compliance mechanisms would break the law.
- Regulatory restrictions on the public usage of an input it is blended with, such as PII data like raw mobile network telemetry. Proper compliance mechanisms are often post-derivation and outside the derived database. Again, making the underlying database available without the compliance mechanisms would be breaking the law.
- OSM is the tail wagging the dog for many of these applications. Trying to make the licensing and compliance of complex applications revolve around OSM's requirements is unjustifiable effort. It is the same reason no one adds a little bit of useful GPL code to a million LoC code base even when it would make sense in the abstract.
This is a long-term problem for OSM in that applications are increasingly about getting intelligent answers out of spatial data systems, which implies the above analytical data models, rather than a map that a human is still required to interpret. Commercial cartographic database providers are usually more than happy to write you a license that conforms with the legal and practical requirements of your analytics application.
I realize that "funding" is an obvious answer, but I'm not real sure what that is supposed to look like. I also don't think the generic answer would convince proponents of the share a like terms (I'm personally ambivalent about share a like, but it's clear that a big chunk of the OSM community values it).
The previous license change was a lesson learned.
Not exactly. The licence can only be changed to a "free and open licence" after a 2/3 vote of active mappers.
http://wiki.osmfoundation.org/wiki/License/Contributor_Terms
If you have never contributed to OSM, the process can be pretty easy:
Go walk/hike/bike the trail so you remember what it looks like, perhaps taking notes and recording a GPS trace of it.
Go to openstreetmap.org, make an account, move the map to the area where the trail starts, click "Edit".
If you recorded a trace, drag it into the editor so it can show it as a reference (hopefully it's a GPX, it might be necessary to convert it).
Using the "line" tool, trace the trail, combining the information from the GPS trace, notes and imagery. Make sure to select the finished trace and mark it as a walking/biking/hiking trail. Then click save.
So there is a lot to do, but none of the individual steps are hugely complicated.
The end goal for this is printable PDF maps, with good data ending up in OSM one of the stepping stones.
a) For planning routes in advance to then download to your smartphone and follow using the app (as mentioned elsewhere OSM was definitely the best mapping layer to use for this both as a resource and as a mechanism to feed your own updates back in).
b) For creating tracks from the routes you've actually cycled which can then be uploaded to the website for later examination.
c) In addition to b) you can annotate your uploaded track on the website with actual pictures from the journey round. These can be from a number of sources (i use flickr) and it will either time sync or gps sync them to the route so you get a nice one stop for looking at previous trips. This has the added benefit you can just take pictures as your going round to then refer back to when fixing OSM without having to worry about making notes as to where it was etc.
Has anyone else used this app or anything else to do this as i'd be interested if there's any alternative ways.
Example annotated track from Hay Bluff: http://my.viewranger.com/track/details/MTIzMzAxNg==
(It's also knowing that all of the local MTB trails that I've surveyed / mapped are now properly on Strava.)
https://strava.zendesk.com/entries/20965883-Elevation-for-Yo...
That being the case, perhaps I should write the program I was thinking of writing. Even though GPS elevation data is really, truly horrible I think you could probably use statistical techniques to get good data with enough GPS traces. Even routes that are mildly popular can have thousands, or tens of thousands of rides per year. Each one of those rides on Strava has GPX data associated with it. I'm sure it would be more than enough.
Unfortunately, I have about a million other projects on the go :-P
I'm particularly bothered by this in the case of MTB Project which works with IMBA, a non-profit, yet doesn't share the data back with the greater world.
It's really nice watching the confluence of open source technologies making a bigger and more powerful wave.
- Get map data from OSM
- Use QGIS for analysis, feature extraction -> export as GeoJSON
- Write GeoJSON to MongoDB instance
Geolocation "works" out of the box thanks to Mongo's geospatial queries and the MongoDB client for Node. Overall, it was an extremely smooth experience, and involved the use of open-source tools from start to finish. This would have probably been impossible to do for free even just a few years back.
There's no regular checkup of the data if it's still valid. Reverting is extremely hard and there are no checks for changing the data. The multipolygon model for areas can be extremely complex and is the cause that many regions aren't touched at all anymore. In many regions it's just an import desert where already open data gets relicensed under the more restrictive OdBL license. Those areas also only rarely get touched by mappers.
I'm working on a project to create a parcel layer on top of OSM, to open source ownership data for all parcels in the US. This is relevant to my interests!
The in-browser editor iD is really easy to use, however currently does not work with IE.
Let's take the first identified weakness: regular checkup of the data.
The easiest way to check this for highways is to run distance matrix calculations. Take 500 points in a given area, and calculate a distance matrix from them (OSRM makes this trivial: https://github.com/Project-OSRM/osrm-backend/). Store the result.
The next day, rerun the same matrix on the latest data. Compare the result against yesterday. If anything has changed significantly, the data has either been broken or improved. Flag this up so that it can be reviewed.
Repeat ad infinitum.
A really simple task for someone with the time and hardware, and it'd make a big difference to OSM. I don't got the time myself, but am happy to help/advise anyone who'd like to take it on.
http://osmose.openstreetmap.fr/en/map/
There isn't really a "central" place to flag such errors.
It is true that very large multi-polygons can be hard to edit however they tend to be exclusively used for modelling large landuse areas and administrative boundaries, and tend to have no impact at all on adding and editing roads, buildings, POIs and so on.
Also, OpenStreetMap provides the data, but Google aggregates the data, which makes very cheap to add/search a lot of information (ex: shops) available from on-line sources (such as web sites). I don't expect OSM to have the same amount of information any time soon.
It looked extremely good. For example they automatically extract signs from streetview and overlay them on the map. They also had a kind of fast streetview fisheye thing. Hard to describe but it looked excellent.
Scout (part of telenav) has a project to recognize traffic signs from free street view videos (dashcams, mapillary photos). Street lanea, exit signs on highway, companies are already working on that.
Mapillary (mapillary.com) offers StreetView similar capabilities, including sign detection and can be used together with the desktop OSM editor JOSM and the in-browser editor iD.
The maps themselves are very good.
Could this model be built automatically from existing OSM data, or would someone else have to manually add some kind of additional hierarchy information? Has this been studied from the computational linguistics side as well as the geospatial information side?
I wonder if GP is talking specifically about on device apps, Photon needs a pretty big index (31 gigabytes compressed):
Just played with Photon on their website: It's better but still far away from Google's capability. Like it or not, but Google is very, very good with search :-)
Naturally comparing with google is a bit apples and oranges, on the one hand a company that purchases essentially all its data and spends billions on its mapping department, compared to the OSMF with a annual budget of roughly $150k. So it is not a surprise that some things take a bit longer with OSM, but it is tortoise vs. hare....
The other problem is OSM tends to have good street maps, but very limited door-to-door coverage.
- Different names for things in different parts of the address because of different languages? (What if two different parts are in different language, e.g. giving a Belgian city name in Flemish but the street name in French? Giving a Swiss address with a street name in French, city name in German, and canton name in French+? Giving a Russian address with the street name in Russian but the city and oblast name in English, possibly using different scripts?)
- Landmarks and points of interest, maybe with separate colloquial and official names? (Tower of St. Vincent? Torre de São Vincente? Torre de Belém? Tower of Belém? Tower of Belem?)
- How do you search if an address component is omitted (like leaving out a city in a U.S. address but including a state or a postal code)?
- How do you handle free-form text searches that might omit address delimiters, or include postal codes in a locally nonstandard order, or omit or include official postal designators for the components of the address (for example, Brazilian addresses might include the word "CEP" before the postal code but could be given without explicitly mentioning that the postal code is a postal code)?
Is there a good book or web site or FAQ about geographic name matching that I could look at to get a sense of what standard answers to these sorts of questions are?
+ In Switzerland, the same city could theoretically be referred to as "Fribourg, [Staat ]Freiburg" or "Freiburg, [État de ]Fribourg". Well, of course a human user might always search using any combination of languages with which they're familiar, or even ones they're not familiar with that they just copied and pasted from somewhere.
OsmAnd~'s interface for this is absolutely terrible.
A corpus of address searches to test against would be really helpful for this kind of development.
I suspect there is a good book or web site or FAQ about geographic name matching, but you have to be a Googler to see it.
https://github.com/geo-data/openstreetmap-tiles-docker
Note: a bit finicky.
Is it possible to calculate directions/routing locally too?
(and sometimes what the community decides, e.g. adding things that can't be seen anymore, like old tram lines, is generally frowned upon in main OSM and requested for something like Open Historical Map instead)
From a navigation standpoint the devil is in the details, like flow speeds and turn restrictions, which is where we see the majority of our issues (flow speeds significantly impact how you treat traffic and turn restrictions of course impact routing).
Often foreigners get confused and see things that look like street signs, but which are actually signs for the chome (kind of a big block). They get very confused when they see another such sign on a different street going in a completely different direction :-)
Anyway, Japanese people generally don't put address numbers on their houses. Instead they will put their name. So if you want to find an address for someone you don't know (or a business) it is really hard. For businesses that have to deliver things, it's really important to be able to find the address. So the public database is really useful. These days everybody generally navigates by GPS. You put the address in and the GPS unit will take you to the right place. Most navigation units will even be able to navigate from a land line phone number (as the phone numbers are pre-assigned to the address).
Address searches are possible though, as ever, it depends whether contributors have added those addresses to OSM.
That's one thing, another is that addresses are quite often duplicated. There's no automatic check for that and such duplicates exist for years.
[1] - https://help.openstreetmap.org/questions/45825/how-can-i-geo...
I've backed the statement that OSM is inconsistent by saying that sometimes streets in OSM contain full name and sometime only most important part (and that makes Nominatim unable to search for addresses).
https://mapzen.com/blog/the-world-is-yours-announcing-mapzen...
I think you'll find that OSM is great for more than just simple stuff. In fact using and contributing to it is a million dollar industry.
I fixed a bunch of stuff wrong with my home town and found it pretty gratifying
I also took a look at it to figure out why it was showing up that way. They look like they are the same magnitude of thing (city) and they both have population. I'm wondering if it's actually showing Camden county?
Mapbox does a great job of fixing things with out relying on users to do it for OSM! Just drop a pin and type out the problem and Mapbox's data team will likely fix the problem! These changes go right into OSM.