Hacker News Reader

Comment by andrewljohnson | Hacker News Reader

This post seems half right, half wrong.

Wrong that OSM should be a service provider and not a database - I think it should be a database, and corporations are good for OSM. The post calls for OSM to essentially become Mapbox. It's insanely expensive to be Mapbox, and Mapzen just failed trying to be a less corporate version of Mapbox. And then the usage policy part of the article is moot or at least not that consequential if you don't believe OSM should be a service provider.

Right: Moderation tools need to be created/fixed. New mappers need to be encouraged/onboarded better. Vandalism is a problem. OSMF culture is toxic. Hidden gatekeepers are toxic. Imports need to be supported. Bots need to be enabled, and we need to consider people who contribute code/tech/imports first class OSM contributors and not just glorify the individual mapper.

Probably right: OSM needs layers, or even a more sweeping re-architecture to allow better versioning, moderation, better tooling, and easier understanding of the data.

Probably wrong: Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.

View on HN

> The post calls for OSM to essentially become Mapbox

The problem is that most companies using OSM are using Mapbox. This leads to a few problems, including attribution, which I decided not to touch on, but is a serious problem that Mapbox doesn't address sufficiently.

If they could use OSM directly, they'd be engaged with OSM directly. They'd care about OSM. Right now OSM shoves these users away and makes them into customers of some third party (maybe Mapbox, maybe someone else) instead of caring about OSM itself.

> Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.

How about letting users experiment and seeing what cool stuff they can come up with?

> The problem is that most companies using OSM are using Mapbox.

Is that even true? And is number of companies even a good metric? Based on revenue/market cap, the bulk of companies use OSM directly.

The world's biggest company (Apple) uses OSM directly. Telenav uses it directly. Don't Tesla and Uber use OSM directly too? And we use Mapbox at Gaia GPS, but we also use OSM data directly.

> How about letting users experiment and seeing what cool stuff they can come up with?

In my experience building a company, if you focus on too many things, you get everything wrong. This smells to me of being not on target for making OSM as strong as possible, even though it has alluring upside.

Tesla is a MapBox customer.

https://electrek.co/2017/07/03/tesla-map-navigation-open-sou...

In the post, he says this is ok for some (implying the giants you mention) but it discourages smaller companies. That second (and likely much bigger) set of people use Mapbox.

I mean, I realize hosting a service is expensive, but can't there be an open source set of service APIs that's self hosted? That way you can chose to get a full copy of the data (and be responsible for pulling updates) and the API, or just pay someone else for access to their hosted version of the API?

  can't there be an open source set of service
  APIs that's self hosted

OSM does have open source self-hosted options.

Unfortunately, the developer experience is much more painful than using Google Maps.

With Google Maps, if you can add Javascript to a web page you're basically good to go. Data updates, software updates, server redundancy, server support and security, all taken care of. Even if you're exclusively a front-end dev, you're making a static site, and you've never heard of Linux.

Compared to that, self-hosting OSM tile generation requires a bunch of extra knowledge, equipment and maintenance.

Is it something that could be containerized? Maybe not as simple as a JS embed, but deploying a self hosted docker container is far easier than installing most things I've encountered.

With vector tiles one doesn't have to containerize anything - just shard them onto a bunch of servers and server them over https. Put an accelerator/cache in front and proxy to the desired region backend.

With image tiles it's a hurdle. You have to build 18 zoom levels of tiles. The most detailed layers have an astronomic number if tiles. You also need a huge amount if storage.

http://tools.geofabrik.de/calc/#type=geofabrik_standard&bbox...

Did I mention the data being huge? What's the point of having 15Tb container, which takes forever to deploy? You can certainly containerize the services, but there's no benefit containerizing the data, since you have to replace it all at once rather than synchronizing the differences.

How intensive is it to rasterise on demand?

Depends on the degree of complexity of the tiles. We tried it (using tilecache or tilestache and mapproxy) and it becomes unresponsive when you rasterize several tiles (9). It can take several secons for a busy tile. If the user gets impacient like I do, zooms in and out and pans the map, it freezes to a halt. It's doable over clear water and maybe on the highway, but not in busy cities, at least not with hundreds of users requesting on demand tiles and without an insane amount of rendering resources. My impression is that it works if you're the only user and a really patient one at that.

Why not use vector tiles? I can generate a vector tile package of the whole planet in Esri .vtpk format and it takes up around 30Gb. A map service can be published from it easily. Takes me about 2 days for initial translation and about 1 week ti generate the vtpk. Neither of these have been optimised for speed. I have an old Galaxy S3 upstairs running an app that displays this vtpk - so I can have the entire world down to house level on my phone.

> you can chose to get a full copy of the data (and be responsible for pulling updates)

Even if you could make the general setup and maintenance incredibly easy somehow, just syncing the map data from osm regularly would require decent hardware: the extra cost of each dev doing this themselves just doesn't compare to using a central tile host.

On top of that, if every dev was regularly updating their own OSM data, OSM would have a lot of extra strain on their data download servers to deal with. Not sure if this could be comparable to the hardware overhead of providing a tile hosting service though.

> just syncing the map data from osm regularly would require decent hardware

We need servers with at least 24 CPU threads and 256 GB RAM and moving the data into PostGIS, region by region takes a week or two. Plus you manually have to fix the process when it just spits out an error and aborts after 15 hours of churning.

Have you tried imposm? Your process sounds magnitudes too slow.

Agreed. I can load with osm2pgsql on my home PC (AMD FX 8370, 32gb ram, 1tb SSD) in a couple of days, and that's generating 78 output layers. imposm is supposed to be faster still.

> just syncing the map data from osm regularly would require decent hardware

You can get country level extracts, which can be updated every day. Nearly any server which can import OSM data is able to keep it up to date.

Yes, you can do just that.

> Probably right: OSM needs layers, or even a more sweeping re-architecture to allow better versioning, moderation, better tooling, and easier understanding of the data.

I have a hobby site about railways [0] I'm picking up again after a decade of inactivity. I wanted to display a route map of each railway, with clickable stations and the route highlighted.

I can not work out how to do that. OSM has the route and station data (there and in Open Railway Map [1]). I expected to find a layer called 'Railways' with sub-layers, one for each railway, and I could export the data and highlight it on another map.

Instead I have raster tiles, and no way to highlight the railway I'm talking about. I know the data must be there somewhere but how on earth do I get it?

[0] https://www.narrow-gauge.co.uk [1] https://www.openrailwaymap.org/

Investigate Overpass and in particular Overpass Turbo.

And really, any strict layering system in OSM would quickly just fall apart into debates of exactly what belonged in each layer. The range of detail captured by OpenStreetMap users in different areas is really far too nuanced to work crammed into coarse, crude "layer" classifications.

How can that be the case? Even with a range of detail, the points on a map have to represent one of a fairly limited selection of types. Even if you keep it high level like "buildings", "roads", etc. it's still a massive improvement over having no layers whatsoever.

> the points on a map have to represent one of a fairly limited selection of types

Absolutely not. Not only is the selection of types huge and full of shades of grey, but an element can be tagged as things from multiple different categories at once (and this is a wonderful feature). Take a browse of the OpenStreetMap wiki sometime. OpenStreetMap data is far richer than a layer model could cope with.

However, let's not confuse this with an editor having a layers-like interface feature to help the user focus on a particular subset of the data. JOSM for instance does this very successfully with the "filters" tool, which I use heavily.

In OSM objects have zero or more tags, a set of key/values (with both being strings). That's all. With that, the OSM community has come up with lots of ways to classify and tag things.

e.g. the railway nerds have very detailed ways to tag railway infastructure: https://wiki.openstreetmap.org/wiki/OpenRailwayMap/Tagging

The type of data you want for routes is probably in KMZ format. I took a few minutes looking around openrailwaymap and their wiki but nothing immediately obvious stood out.

https://catalog.data.gov/dataset/railroad-kmz

A brief google search came back with this; I haven't inspected the data or even tried to use it to see what it contains, but the page description indicates it may not be exact routes you are looking for-but it's a start. Implementing the data is a bit more work in order to display route maps, something like the previously mentioned MapBox does this very interactively.

My email address is in the profile, feel free to shoot over an email if you'd like some help with it. Depending on the complexity, I can probably devote a few cycles this weekend to show you a one-off map with railway routes, and show you how it's done.

Is KMZ really still the best way to go?

Good question.

I'm only a hobbyist with GIS, so I can't really say or speak to how practical of a format KML/KMZ are in production.

But that's a really good question.

I work in GIS.

KML/KMZ is decently useful as an interchange format and simple rendering but almost all web GIS software has better support for GeoJSON. It’s richer and easier to work with.

I’ve never worked on a GIS system that used KML/KMZ in prod for rendering. It’s usually shapefiles, PostGIS Geometry, ESRI’s proprietary GDB feature, WKB/WKT or GeoJSON on the backend being served up to clients.

I have used it as a lightweight way to deliver simple data in the past though for use cases where there was no proper client renderer (we would just tell the end user to download google earth and open up the file)

Thanks for the offer, I got 75% through writing you an email and finally managed to get some data exported via https://overpass-turbo.eu/, so I'll work on that for now. Thanks though!

OSM is essentially a database of points and lines, each with a (loosely defined) soup of tags attached to it. Highly flexible (annoyingly so, if you've got a single focus area) and not in a "layer" format.

If the data is in OpenMapTiles, which are vector files from OSM data, you may have more success styling it in the browser.

https://openmaptiles.org/

In some sense there is a layer, features related to railways will all have a railway tag (or be listed as a member of one of those features). There's some description here:

https://wiki.openstreetmap.org/wiki/Railways

In the sense of being easy to retrieve I guess it isn't a layer. Global railway data will be pretty huge, so Overpass-API may not work out if that is what you want. In that case, the strategy is to grab a planet file (a snapshot of the db) and filter it locally:

http://osmcode.org/osmium-tool/manual.html#filtering-by-tags

And then you probably have to translate the resulting data into a format that you would use for your map.

As to why you have to do that work instead of just pulling layers, it comes down to the core data model being focused on editing and there being many different end use cases (making it difficult to anticipate and serve each one).

Here’s something similar I’ve built: https://stereo.lu/charely

Thanks, that was fascinating to read (even allowing for my poor french). That is the kind of data I'm looking to get out.

You might be interested in this article from Alessandro, the guy who makes Spatialite: https://www.gaia-gis.it/fossil/spatialite-tools/wiki?name=gr...

It talks about the errors in Open Street Map data, and the consequences of that for railway routing.

Check out the overpass <https://overpass-turbo.eu>. You should be able to query for the data you are looking for.

I'm currently doing something very similar for my school project and it's been quite a journey, the OSM ecosystem sure isn't simple.

My current solution is based on OpenMapTiles and uses their stack for creating, and subsequently rendering, vector tiles. First, I obtain an extract of OSM data via the Overpass API. Then I convert it from XML to protobuf, as it's a much more efficient format. Afterwards I import it into a PostGIS database using imposm3 and define some SQL functions that transform that data. Then I generate some vector tile definition from the OMT format into tm2source format. After that I finally generate the vector tiles in mbtiles format using tilelive, server them as GeoJSON using tileserver-gl and style them using maputnik and mapbox-gl styles.

There seems to be some bug with imposm3 and relations, because I can't get them to import. Or I'm just not doing it right.

Wow that is complicated! Thanks for sharing, it's good to know how time consuming this could be if I take it further.

It sounds like Overpass is the way to get the data I need - once I figure out how to reference it in Overpass (network/relation details).

You make the point OSM should be a database and not a service, but that’s a false dichotomy as nothing inherently prevents it from being both.

The main point against a service seems to be that’s it’s an expensive thing to do, however I don’t see how we can draw any conclusion from that without weighing the costs against the upsides.

I don’t see how the expense argument can be made with confidence without first establishing some basic data points. For example:

1) How expensive is expensive?

2) Have the benefits and advantages been throughly explored ie what’s the value proposition?

3) Given this value proposition, what level of fund raising success would be plausible?

As an aside, the ineffectiveness of leadership and how it is seemingly blocking progress is quite striking.

Off the top of my head, I can’t think of any actively developed software with a non-deprecated api that hasn’t changed in so many years.

OSM has gained countless new capabilities since moving to API 0.6 in 2009. World-class, super-fast routing. Beautiful vector maps. A much easier raster map API. A powerful and performant query language. To say that progress is being "blocked" is not at all accurate.

Sure, these aren't provided by what's formally known as the "OSM API". They are provided by OSRM/Graphhopper, Mapbox GL/Tangram, Leaflet/OpenLayers 3, and Overpass respectively. Because that's, deliberately, how OSM works: it's the centre of an ecosystem, not a single service provider.

The OSM API is for the use of map editing programs. It's an implementation detail for the authors of iD, JOSM, Vespucci and a few others. It isn't something that anyone who isn't writing an editor needs to concern themselves about; and I honestly haven't heard many comments from editor maintainers that they want the API to do things it doesn't at present (though, Bryan, Dirk, Simon, do jump in and correct me if I'm wrong).

Do you have any connection to any company involved with OSM tech, just so I understand your perspective?

Mentioning OSM has gained functionality is irrelevant, the article’s point is the API which serves a different purpose. What other companies offer is irrelevant that’s not OSM.

Your other points seem to attempt to diminish the importance of the API, yet the state of modern software has become such that almost any significant software project can benefit from a rich, well implemented and maintained API.

Not to mention maintaining a high quality API often in and of itself leads to improvements in the overall architecture of software, due to how it encourages careful thought about the purpose and organization of the system.

The "OSM API" is an editing API, for editing the map.

The OSM ecosystem has gained some new, much more powerful APIs, which Doctor_Fegg mentions. OSRM's API, GraphHopper's API, Overpass's API, these are all better than the APIs that were available in 2009

Again I would ask, how does that invalidate the article’s premise that the core software API hasn’t changed in ~10 years?

What other significant, widely used software, depends only on the “ecosystem” for APIs development, while having no active API efforts in the core project?

I can think of none. Maybe some examples exist somewhere, but at the least it’s unusual, especially given the overall context of the article.

The core editing API hasn't changed in years. HTTP/1.1 come out in 1997 and HTTP/2 in 2015. Was web programming in 2014 the same as web programming in 1998?

Have you ever designed and implemented a modern api? It doesn’t work like that.

HTTP itself is fundamental and a different level of abstraction. We would not want monthly updates.

A closer analogy is Wikipedia (see wikimedia api). It hasn’t had the fastest evolution, but it makes OSM look like the stone age. You could look at the github api as a project that’s invested more frequently.

And it’s not just because these are projects with vast riches (even though WP’s multi-million budget is actually tight). There are countless smaller projects that keep a well maintained API. I just list these because they are well known.

The role and importance of APIs in software has changed dramatically over the last 20 years. It’s true at one point it was more of a luxury and just wasn’t as central to many projects.

Those days are far enough in the past and so anachronistic that a ~10 year stale API at this point should be considered architectural malpractice.

This discussion might be more productive if you laid out what you believe the OSM API being discussed already does and what you think it should do in addition to that.

And then you could lay out why those things should be done centrally rather than on separate servers. For example, there is a sophisticated data query api https://wiki.openstreetmap.org/wiki/Overpass_API that keeps up with edits within a minute or two but is free to lay out the backing database in a way that is optimized for queries and people can run their own instance and so on.

Oh yes, I agree. In fact, I’m taking a close look at the architecture, organizational efficiency and results to form an opinion of the bigger picture. Indeed, I haven’t, scratch that, can’t, even assert an opinion on the article as a whole without doing so.

Then how is this productive you ask? Because for now I’m only drawing much smaller conclusions, which also generalize to most actively developed software. Rich, well maintained APIs tend to be useful. As of yet I see no real explanation of why that doesn’t apply to OSM.

To start, not sure why you mention a central server vs. “separate servers”. SaaS users shouldn’t care, or ideally even need to know whether something runs on 10 or 100 servers.

Maybe instead of separate servers, you meant different companies? Knowing which organization is the provider of each service is paramount. For each provider it’s also important to know something about their business model, reputation, roadmap, and the pros/cons of using multiple providers vs. one stop shopping. It’s important enough that companies like Gartner are paid a truckload of money by SaaS users to analyze these details.

So back to my original point: Most significant and active projects benefit from investing in rich and well maintained APIs.

Take that fact, and add in a domain expert taking the time to write an article asserting it’s indeed a problem relevant to OSM. One person’s opinion is not always a smoking gun. But combined with the fact a 10 year old stale API is so unusual, and the fact that what other companies do is a red herring argument being made, at minimum it sure seems like a bright red flag that warrants better understanding of what’s going on over there.

I mention a central server vs. “separate servers”. because that's pretty much the only way you could come to the incorrect conclusion of 10 year old stale API.

There's a particular service running on some OSM servers that provides data to editing software and receives changes from editing software and applies those changes to the master database. The primary version number on that api is 10 years old. It has still evolved over that time. But it is not the entirety of the OSM API, it is the OSM editing API, which probably benefits from being at least somewhat stable.

I mean, I guess the basic data model could be changed all the time so that all the tooling had to chase after it, but the only big proposal along those lines is to add an explicit "area" type which would simplify some things (it wouldn't really create room for any new capabilities).

Take a look at the vector tile stack Mapbox has put together if you want to see a new way of working with OSM data that has come about over that same 10 year period. Mapzen and OpenMapTiles have also worked on vector tile stacks.

Mapzen tried to be a less corporate version of Mapbox but the quality of their data/tools was just not there.

I had to implemented a specialized geocoder/administrative boundaries referential last year. Mapzen was really compelling, the documentation about how they handled hierarchical data was clear and interesting but execution was lacking. I am French, handling European data was important and I quickly hit several pain points:

- Their github repository organization was a mess. A lot of small ones without a clear map of which did what. - The vagrant setup to experiment locally was unmaintained - I found a basic unicode normalization issue in the geocoder several hours after playing with it. - Manipulating their dataset via git-lfs was a massive pain - Their geocoder was nicely presented but relations to administrative levels was done with neightbourhood search to administrative area centroids, instead of (even approximated) geometries. - The data was not as comprehensive and up to date as OSM one, at least in France. For instance, France second level administrative areas (Régions) were redesigned at the beginning of 2016. They were defined in OSM but progress seemed fairly slow in Mapzen.

Now, I appreciate Mapzen was a very ambitious project and handling cartographic data at this scope and scale is a difficult job, but as pure data consumer, it was not comparable to OSM.

I eventually used OSM data and while their documentation/tools are quite a mess too, wiki style, it is workable. Datasets are large but there are a lot of way to retrieve them efficiently. Their non-protobuf binary format hits a sweet spot between size efficiency and ease of processing. And at least in Europe, the data is fairly good, at least for my requirements.

> Probably wrong: Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.

Maybe I'm misunderstanding the whole domain, but if OSM would get layers, that would lead to better composability, then wouldn't the question of transient/frivolous data kinda become moot? The way I see it, in that scenario I could make and host my own whatever pokemon layer and OSM proper could provide the base layer, and then they could be easily merged at usage point; whereas based on the discussion now without layers you'd need to essentially make hard fork of the OSM data to make your own pokemon map?

Perhaps I'm misunderstanding your goal, but how could this not be achieved with a separate database, and overlay your pokemon on-top of the OSM map?

> Imports need to be supported.

I don't really agree. I have seen so many wrong imports and I have seen imports demotivating individual mappers.

With a good data source an import can be a good start, but it's not a silver bullet.

I agree to your other items, though.

We wanted to contribute forest data on Africa back in 2015. It wasn't possible. We'd have to add it via JOSM or the web based editor. So we gave up since it was a no go. Now we just discard the OSM forest data and add our own when updating the map data.

Are you not violating the licence in this case? Or do you make the new dataset available under the same licence? I think the licence is one of OSM’s big problems.

> It wasn't possible. We'd have to add it via JOSM or the web based editor.

That's strange. What were you trying to do? If you convert your data to OSM's relatively simple XML format, you can open that file in JOSM, and just press the upload button. That's what most people do with imports. It's possible to communicate with the OSM API directly and make API calls, but that's essentially what JOSM does. JOSM also has a validator, which can spot some problems with your data.

Transit data should probably be supported in a meta-sense. A machine readable bounded region with annotations about where to fetch transit data and how to parse it should likely be added. That way the OSM database links to external data sources.

> Probably wrong: Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.

Isn't all of the data in OSM transient to some degree?

What's the threshold for when data is too fleeting to be worth storing? Days, months, years, decades?

Test

Reply on news.ycombinator.com