2024: The year of the OpenStreetMap vector maps | Hacker News Reader

2024: The year of the OpenStreetMap vector maps

blog.openstreetmap.org

345 pointsRicoElectrico82 comments

Way overdue. OpenStreetMap's website at openstreetmap.org is its calling card, and for the past few years the default style shown (called Carto) has all but stagnated in development. Accepted features like highway=busway (introduced three years ago) are not rendered there because the maintainers can no longer be bothered, or dislike the tag personally despite broad community backing.

What worries me for this new effort is that Paul Norman is one of the two remaining Carto sometimes-active maintainers who refuse to merge contributed PRs or even provide alternative minimal support for features like highway=busway, leading to awkward gaps on the baseline map shown on openstreetmap.org.

I would love to be surprised in a positive way about this new effort, but I'm not holding my hopes up. Thankfully OpenStreetMap can be thoroughly useful in apps like OsmAnd and OrganicMaps, and the tile-based Tracestrack Topo layer on openstreetmap.org is getting quite decent:

https://www.openstreetmap.org/#layers=P

Apparently even Paul Norman is tired of contributing at openstreetmap-carto.

Honestly of the maintainers imagico is the one I'd say is impossible to reason with.

Oh absolutely. I have given up on engaging with him, it is pointless (I am jdhoek on GitHub, and have contributed various commits to Carto in the past).

While I can understand that being a maintainer for a project like Carto can be taxing due to the high amount of feature requests which are often not suitable for the general purpose map, lack usage, or are otherwise unclear in their definition, I find that the way Carto has been maintained for the past few years has been harmful to the OpenStreetMap project in general.

I think Christopher Hormann (imagico), despite his contributions in the past, is actively harming OpenStreetMap with his maintainership and (lack of) actions. Although to be fair the OpenStreetMap Foundation shares some of the blame here by shrugging 'not our problem' when the state of the default layer on openstreetmap.org is, repeatedly, pointed out, and letting this state of affairs persist. The typical response of 'Oh, OpenStreetMap is a do-ocracy, haha, if you want to change this, just create your own layer!' does not befit a project which has achieved so much as OpenStreetMap.

I know nothing about the specific issues here, but as you mentioned do-ocracy: Would it be viable to fork carto and merge the good stuff that has been originally rejected, then asking openstreetmap.org to change their map renderer to the fork? If it's objectively better and has broad community support, that should be in the best interest of OSM.

(once again: very little knowledge of OSM, so maybe that's overly naive)

Absolutely doable but not really worthwhile right now. Vector tiles open up many more possibilities for custom cartography and I have every confidence Paul’s work will live up to this promise. osm-carto is essentially a dead end cartographically and technically.

Possibly, but in reality, not really. Without buy-in from the OSMF (the foundation) this could result in a lot of effort and time ultimately wasted due to overlapping roles of people concerned in the Operations Working Group.

Most people with the knowledge and capacity to do this have spent their energy on other initiatives not hosted directly on openstreetmap.org instead (like OpenStreetMap Americana destined for the US chapter's OSM website), or contributing to OsmAnd or OrganicMaps.

OSMF chairperson here. I share your frustration. This is us doing something about it!

Yes, it would be quite important to improve the website if the idea of having it added to Firefox' start page comes through [1]. Polish is important for first impressions :)

Anyone from Mozilla/Firefox here?

[1] https://community.openstreetmap.org/t/idea-ask-mozilla-for-o...

Slightly offtopic

I have OsmAnd ( https://github.com/osmandapp/OsmAnd ) on my phone, download the basemaps, download my (small) country data (both sourced from open street maps), and with an app + ~1GB of data, I get the maps and full navigation within my country, POIs, etc., and can add other countries when needed.

Is there something similar for a PC? I can download data from open street maps, but then I need postgres, postgis, a tile server and styles and apache running just to generate the tiles. Is there anything portable (short of running osmand in an android virtualbox) for offline navigation on a linux pc? QGIS can display vectors, but I wasn't able to easily style the data... navigation is a no-go there too. anything else?

There is a beta of Organic Maps for Linux. Check out bottom of this page: https://organicmaps.app/

I would still want to to know if I can selfhost something of similar, low complexity in my home network.

You can download an extract of your country from Geofabrik, run it through Tilemaker (https://github.com/systemed/tilemaker) to get a nice mbtiles file, and then use the built-in Ruby server to give you something you can load in your web-browser locally.

Post author here. Geofabrik has even started providing extracts in the Shortbread vector schema, the same schema we'll be using for our vector tiles.

I loaded OSM into my Garmin and have used it to drive around the world. No matter the country from Sudan to the Congo to France, the USA and Namibia, the maps have been precise to a T, and I'm staggered at the level of detail and completeness.

Completely free, too.

KDE Marble and GNOME Maps both use OSM, but I don't think the latter uses vector maps. The former does navigation.

As an aside, you can likely run OsmAnd in a container without having to use Virtualbox at all.

What about: https://marble.kde.org/features.php ?

Not Linux, so it's probably not going to help you much, but the OsmAnd iPad app seems to work just fine on macOS and afaik those run as native apps on M1/2/3 machines, and this particular one feels very similar to the macOS-native Maps app. I guess that's pretty close for those on a Macbook, though turn-by-turn navigation is going to suck with no onboard GPS. I just got it from the App Store to test this and I think I'm going to keep it, it's actually pretty neat.

There was(?) an awesome app Sas.Planet that worked with lots of map providers [0] and provided much more features, than just a map viewer. For example, I successfully used it to create offline satellite maps of remote areas. The team seems to still work on the project, but I wasn't able to find the site I was using a while ago, so giving a link to github.

A few word of caution:

- the app was a bit buggy (but usable)

- might not be well optimized in terms of amount of http requests sent, so some map providers don't like it [1]

[0] https://github.com/sasgis/sas.planet.src

[1] https://wiki.openstreetmap.org/wiki/SAS.Planet

In the late 90's early 00's my family used OziExplorer on various laptops and devices. Our thinkpad 240 had a rough life in the bush and on the water.

A quick look suggests that it has some support for osm data but you would need to check more closely or use other sources.

https://www.oziexplorer4.com/w/

https://github.com/janbar/osmin/

https://github.com/rinigus/pure-maps/ (makes use of OSM Scout Server)

There's a port of OSMAnd, looks like it has stagnated.

https://sourceforge.net/projects/offroadosm/files/

Cruiser (proprietary but free of charge): https://wiki.openstreetmap.org/wiki/Cruiser

If anyone wants to experiment with vector OSM data today I can very firmly recommend Protomaps - I wrote about that here https://til.simonwillison.net/gis/pmtiles

Here's a vector map I built with it of Half Moon Bay California https://simonw.github.io/hmb-map/

Oh this is awesome! Bookmarking for later, thank you for the link!

I'm excited for this! I have noticed that most apps that use OSM data provide offline vector downloads for each region. Are most of these apps building their own vector databases? I assume that OSM themselves providing vector tiles will be game changing, allowing app developers to skip creating their own vector downloads as they could just hook into the live vector tiles. No doubt this will make providing OSM maps a much simpler process.

You often want to build your own tiles for specialized applications. Any map schema is a tradeoff -- if you include every feature in every tile at every zoom level so that the renderer can pick and choose what to show, you bloat the tile file size.

The blog post is very light on details. My intuition is that this is primarily focused on powering the map on osm.org in service of people who edit OSM getting to see faster updates. This would be in keeping with their current raster tiles, which are not meant for use in "real" applications.

There are enough technical challenges in rolling out vector tiles with minutely updates for editors on OSM -- in my opinion, they should solve it for OSM first before thinking about committing to a specific schema for public use by other apps.

Still, anything that incentivizes innovation on vector tiles is pretty exciting. One option that could be interesting: provide "fat" builds of tiles that have every feature, and create tooling to do something akin to treeshaking so people can simplify the resulting tiles to only contain the features they need for their use case.

Post author here! Yes, most apps have to create their own tile or download server at the moment. We’re hoping that our tiles will be useful to the greater ecosystem, in the same way that our current raster tiles get used everywhere.

The nice people at https://download.geofabrik.de provide daily country-level OpenStreetMap extracts in the Shortbread schema, which is the same schema that we are going to use for our live tiles.

Wait I thought these downloads are raw data (osm.pbf). Where are the shortbread downloads? Is that what the shp files are?

If you look at, say, https://download.geofabrik.de/europe/luxembourg.html you'll see a link for "experimental vector tile package conforming to Shortbread schema for use with MapLibre and other MVT capable software".

Seems like they are not available for all regions or countries. E.g. the pages for Europe, Germany and UK don't have such a link. I guess this is related to data and file size.

I recently made a self-hostable web app* that works the same way; the user downloads regions with a point and click, I have the data on S3. I used Protomaps, which is vector library for the browser. I did look into seeing if I could just download the maps straight from Protomaps. I didn't end up doing so, I think it's because they didn't seem to have an API that I felt like I could scrape wholesale.

But also, it turns out that there's a question of how that data is generated. There's a certain schema of what sorts of things show up in the browser, perhaps with styles giving options for how to display them. But which raw OSM data you decide to include, and which visual element it corresponds to, is up to the producer of that data. So thinking of Organic Maps and OsmAnd, I bet they choose slighly different things and probably change it over time.

* https://sandstorm.org/news/2023-12-05-osm-on-sandstorm

The maps on openstreetmap are nice of course. But most websites using openstreetmap don't get their maps from there but from companies like mapbox, maptiler, and others that typically already provided vector maps. Openstreetmap itself has been lagging behind a bit.

Protomaps, mentioned elsewhere in this thread, is awesome for self hosting vector maps. The way it works is that you upload a single large pbf (protobuf) file with all map content and then use lambda functions to extract vector tiles from there. Put a CDN in front of it and you have cheap map hosting. Amazing stuff. I met Brandon Liu, the person that built this, some time ago at a meetup. Well worth checking out his stuff.

There's also https://demo.f4map.com/#lat=47.5933049&lon=12.1748491&zoom=1..., which is a 3D rendering engine filled with openstreetmap data. Pretty nice stuff and be sure to fiddle with the options to change the time of day, turn on the moving ships, etc.

This thing is awesome! In the city I live in (outside the US!) you have special renderings for sky scrapers and even a crane where one is being built right now: https://demo.f4map.com/#lat=50.1134349&lon=8.6709344&zoom=19 Awesome!

Although it's lagging behind. Four is almost finished but you only see cranes here: https://demo.f4map.com/#lat=50.1121791&lon=8.6738090&zoom=18...

This is because no-one mapped the outline of the `building=construction` yet.

You can do so if you know more, go to https://www.openstreetmap.org/way/592118388 and click the 'Edit' button. The currently available 'best' aerial imagery shows the foundations being built.

you don't need a lambda function if it can use the range header to get what it needs, also it is an aggregated pbf on a pyramid of space filling hilbert curve.

Indeed, you can use pmtiles files on a completely static site which is the main benefit for me. I wrote it up on my blog: https://blog.gpkb.org/posts/hugo-static-site-maps/ I have since started making my own pmtiles from pbf extracts using planetiler, though. I should add that as an addendum...

We're actually looking into using hugo for our main website currently.

However, running pm-tiles at build/deploy time for the whole world is a bit expensive and time consuming. The point of using lambda functions is to run it on demand for just the tiles that are needed and then cache the results via a CDN.

Hmm... I don't really understand what pmtiles adds in that setup. I thought you'd need to have a pmtiles basemap in order to efficiently extract tiles in a Lambda function. Or are you generating much small pmtiles files from the pbf on demand? Do you need an initial sweep of the whole world to make this performant on first hit of each region?

Which tech stack are they planning to use? It seems like Planetiler and tilemaker are popular options but do they scale for OSM's needs? Protomaps PMtiles are also very interesting

Post author here! We’re going to reuse a lot of existing bricks like osm2pgsql, postgis’s ST_AsMVT, and the shortbread schema to start with. Tilekiln, the project Paul is working on, is python to orchestrate it all, and regenerate tiles on database updates.

Probably something else that is straightforward to incrementally update (applies to all 3 cases from your comment).

https://www.paulnorman.ca/blog/2022/10/tilekiln-storage/

Thanks. I found this article on Paul's blog which answers my question: https://www.paulnorman.ca/blog/2024/01/minutely-updated-tile...

As a religious OsmAnd user I welcome this. I'll be donating directly to the effort.

But OsmAnd already uses their own vector tiles.

I hope they come up with a nice, clean, standard for a vector tile format.

I haven't spent a lot of time looking into it. But the last time I looked at commonly-used vector tile formats, the one I found was clunky to generate/parse.

(And/or, maybe I just missed the right one. Please correct me if there is something out there that is clean and simple -- and has common library support.)

Mapbox Vector Tiles are, for better or worse, the industry standard. They have good library support and are good enough that no one is rushing to reinvent them.

If you want to distribute sets of tiles to people, PMTiles is the best container format as you can just throw the file on hosting that supports HTTP Range requests (e.g. S3) and add the appropriate plugins to your client library. If you have to support clients which can't do the range requests from pmtiles, you can throw a thin layer in front of it to turn HTTP z/x/y requests into the pmtiles range requests.

There are no good file formats for distributing minutely updates of tiles but there's no need for this. If you are keeping your tiles up to date up to the minute you have a reasonable internet connection and you can just download individual tiles instead.

Thanks!

I just browsed the spec again (https://github.com/mapbox/vector-tile-spec/tree/master/2.1) -- I didn't have as bad of a reaction as I did the first time I looked at it.

I guess the idea of using (Google) protobufs for the "packaging" and then a super tight, custom, byte-encoded format for drawing commands seems... weird to me. But, honestly, it probably makes sense.

Also a post from Paul Normans, mentioned in the parent post, covers some options. https://www.paulnorman.ca/blog/2022/10/tilekiln-storage/

Not an expert, however (lol) - even if the recommendations here are solid - this doesn't feel like a particularly well-reasoned comparison.

IMHO maintaining a large number of files on disk (be it 16m or 1.5b) should be a non-starter (not to mention that zipping them to slightly reduce the count removes one of the benefits of having them in the first place). A database makes sense - as does SOC between generation and distribution if it brings benefit - but immediately writing off SQLite because of write performance[1]? Similarly the arguments against pmtiles feel outdated and ignorant of the developing nature of the protocol (appreciate it's a 2yo post!).

OSM deciding on a format here has potential to significantly shape things to come. Hopefully Paul has progressed his thinking a bit since this was written.

[1] https://blog.wesleyac.com/posts/consider-sqlite

Most of the development for pmtiles has happened in the last 2 years, including a maturing server implementation (http://github.com/protomaps/go-pmtiles) but some key parts are still missing like the ability to decode an archive in native mobile applications. sqlite (mbtiles) already has ~10+ years of integration into the mapping ecosystem so that still works better if you want to move tilesets around to desktop applications and mobile devices.

The situation Paul is addressing is one unique to OpenStreetMap itself, which is minute-level updates of a global scale tileset. This is a use case pmtiles is designed explicitly not to address, where using a database is a better fit.

Appreciate you weighing in directly Brandon! To be clear though, I didn't mean to say that protomaps was potentially a better solution (of course an uncompressed DB makes much more sense), simply that the developing nature of the protocol meant that it's inclusion within the "ecosystem" shouldn't be a closed case :)

Which is to say, distribution of OSM data feels like a large part of the process. Of course there are various bottlenecks / considerations around edits / writes, but in practice surely reads are the bigger vector. I wonder how many OSM-external use cases rely on "minutely" updates or need the full fidelity of the raw data source. Feels like there is a solid case for providing less frequent (hourly, daily?) official updates via official "single-file" formats that could be widely distributed to the benefit of all, and e.g. allow OSM to loosen up their hot-linking policies and ensure continued investment in the chosen protocols.

But mainly I was questioning how a somewhat proven format like SQLite, with its many benefits (interoperability, distribution, etc.), would be so easily dropped from consideration without even a test having been run. Just my thoughts of course!

There is nothing stopping the OSM Foundation from say, offering a complete SQLite (or PMTiles) tileset download on planet.openstreetmap.org, technically, legally or otherwise. Creating and archiving the tiles shown on osm.org would be at least a couple terabytes once you get to around zoom level 16.

The key distinction is "official" - the only "official" data product of OpenStreetMap are its XML and PBF data dumps at planet.openstreetmap.org. The frequently-updated tiles you see on osm.org are "quasi-official", they're created by a separate project called OpenStreetMap Carto. These tiles have a special status within the OSM ecosystem for historical reasons; by virtue of OSM being map data, it should probably show something for human eyeballs on the website.

The design goals of OSM Carto are to show feedback to map editors; the linked vector tiles project is intended as a successor, or at least complement, to OSM Carto. The consumption of the tiles by third-party sites is a side-effect tolerated by OSMF; the general consensus within OSM seems to be that a consumable tile service for third parties is outside the scope of the project.

Great context + points. Thanks!

Definitely not in a position to question the status quo, let alone the OSM community consensus. Other than to say that - even if the focus is currently on internal software / editors alone - it still feels like a massive opportunity. And one that aligns with a number of strategic goals of OSMF[1], specifically around efforts to grow the community, extend the core developer base (and more) ... by fostering a closer relationship between OSM data-derived products / the wider (non-editor) community.

Let it be known I love scope creep.

[1] https://osmfoundation.org/wiki/Strategic_Plan

Yeah we are considering serving tileset downloads there. It should be easier to get OpenStreetMap data for your offline or on-premises map.

The SQLite limitations might also be solvable by sharding the databases (so each sees 1/N of the write volume), tiering them (so tiles that are written most recently are in file A, then file B, then file C), or a combination of these approaches.

I'm assuming the requirement is not really to build a single .mbtiles file, but to have some way for a process to serve the correct tile when given a z/x/y tuple.

What does the pmtiles protocol provide to help update the tiles in place?

That's the challenge with the osm.org feedback map, you need to quickly update the small fraction of tiles that very recently changed, while hopefully not processing the remainder.

It doesn't, pmtiles is designed for tilesets updated at a daily to quarterly frequency, by replacing the entire archive on cloud storage. for minute-level updates serving tiles out of PostGIS is the best current approach, which is the approach Paul is taking for the OSMF project.

Have you tried pmtiles? https://docs.protomaps.com/

Slightly related to OSM, can anyone recommend a good and polished iOS app? I recently tried to use OSM more but the web view on mobile was impossible to use. I was not able to navigate from point A to B with it.

I’ve had good luck with Organic Maps [1] on iOS.

[1] https://apps.apple.com/us/app/organic-maps-offline-hike-bike...

Looks good, thank you!

Having tried multiple times to move away from Gmaps on Android, all the OSM based apps I've tried are barely usable at best (sadly).

Have you tried Organic Maps, mapy.cz and Osmand?

I’m curious why you wouldn’t use the Apple Maps app or something else? Is it to promote open software?

There’s some hiking trails in the city that are only on OSM as they are not “official”, that’s why sometimes it’s nice to have a choice.

I’m back on Google Maps as the POI data in Apple Maps is unfortunately very incomplete (Opening times, reviews, new restaurants etc.) even in bigger cities like Berlin.

Makes sense. POI data coverage varies from place to place.

I like the mapy.cz app (and website) a whole lot.

Have a look at https://www.serverlessmaps.com/ which uses PMTiles to generate the tiles from OSM data. It's also possible to host the PMTiles directly on S3 and use HTTP range requests to retrieve the desired bounding boxes. Under the hood, it uses CloudFront as CDN, Lambda@Edge functions and S3 (see architecture at https://github.com/serverlessmaps/serverlessmaps/#architectu...)

It's a very cost-effective solution as CloudFront has a 1TB/Month free tier, compared to hosted solutions.

OSM is a great project

I don't use OpenStreeMap because their data quality is poor.

That really depends what you plan to use where. Roads and foot navigation is generally quite good, POIs is good to use in large cities. Where do you see OSM data lacking?

Nope. It took three years to update a tunnel in my local area. Even now, other streets I've sampled which less than ten years old are wrong.

What you're saying is that government data is good, which OpenStreetMap has taken. But that data doesn't update very frequently.

> It took three years to update a tunnel in my local area.

The idea is that you can update it yourself?

I happened to be using OpenStreetMaps for the last couple weeks and released a Golang OpenAPI library for OpenChargeMap [1] In that, I released an IPython notebook and used that to visualize the data on Vision Pro [2].

And I did get it working! But it was not near as cool as it could be because of two dramatic events:

OSM Foundation violated the trust (nice way to put it) of a participant, OSMBuildings.org, in the search for sponsorship [3] (four years ago, such lost progress :( ) The other is that LeafletJS’s developer is Ukrainian and can’t advance his project [4].

Please consider those two issues when contemplating donating to OSM due to this press release. While I applaud OSM abstractly, it exists because of a massive amounts of public contributions of IP, not because of $$ from a commercial sponsor.

[1] https://github.com/neomantra/go-openchargemap [2] https://github.com/neomantra/go-openchargemap/blob/main/exam... [3] https://medium.com/@osmbuildings/why-were-not-going-to-suppo... [4] https://leafletjs.com/

It seems unlikely to me that the OSMF did something shady in exchange for the promise of 6,000 Euros from Cesium.

The OSM Buildings folks called themselves "OSM Buildings", using the OSM trademark. They offered a commercial service. The OSMF was OK with this. It seems the OSMF is OK with a lot of things -- including permitting Cesium to use the OSM mark in their own commercial project.

Live and let live, may a thousand flowers bloom, etc.

Rather than reflect on the fact that their name was not really theirs, but relied on the goodwill of the OSM mark, the OSM Buildings folks decided to ragequit and shut down their project, dramatically claiming that their name was "seized by a multimillion dollar company"?

The whole thing is bizarre to me.

I was trying to understand it from both sides and not getting much. Some of your color helps too (like the euro amounts). It all was bizarre and sadly the whole thing blocked some progress.

I had come back to temper my response because for sure this is great work by the tech team.

Some more colour is that the OSM Buildings project didn't even ask permission to use the OSM mark. If they were a community project, this was permissible. If they were a commercial project, they ought to have sought permission from the OSMF.

Given that, it seems pretty reasonable to me that the OSMF's stance was basically: your name is your problem, if you think someone is infringing on it, it's up to you to enforce it.

The OSMF runs on a shoestring budget - something like $400,000/year, vs Wikimedia Foundation, which runs on $170,000,000/year. It would be a real shame if a potential donor thought this particular instance was a reason not to donate to the OSMF.

Leaflet is a separate project from OpenStreetMap - it's just one popular frontend for viewing tiles served from openstreetmap.org, like OpenLayers.

I don't believe the OSM Foundation has ever been involved in Leaflet development, as Leaflet development was supported in the past by commercial vendors of OSM data like CloudMade and Mapbox.

Maybe faux pas to reply to my own comment I made further connections with this.

I literally had cloned and was fixing OSMBuildings.org because, among other things, it wasn't clamping values correctly [1] -- it's why my charging markers are white. But I felt conflicted doing that. I was happy with what I spiked with OSMBuildings and started exploring commercial services like MapBox and a few other OSS projects.

Operating projects and companies and organizations is not easy. It sucks what happened there. I'm not a hater and actually looked to Patreon or similar Paul Norman -- and maybe the 'vector tiles' earmark literally means that? But, I'm not happy with that previous behavior of the organization...

So it's great the OSM Vector Tile service will be dramatically improved...

If any of us start doing something interesting with, will we get jacked up like OSMBuildings? Is that the stewardship of this donations? That's the rub.

[1] https://github.com/OSMBuildings/OSMBuildings/issues/230

OSMF chairperson here. The osmbuildings situation is unfortunate. They never asked to use the trademark, didn’t participate in community discussions, and used OSM + a very generic term for their project name. Cesium asked for our permission to use the same words, and simultaneously, but not in exchange for any permission, joined as a member to support the foundation. That money went to paying for the infrastructure that runs the project.

We never asked osmbuildings to stop using the name. We’ve never granted exclusive trademark permission to anyone - if you wanted to launch a product called “neomantra OSM buildings”, we’d find it cool.

If you start an interesting project, pick a non-generic name.

In all honesty, I find 3D buildings such a gimmick anyway. I'd rather see a flat map with less clutter.

Seeing building heights is an important way of knowing which part of an unfamiliar city is the central/business district, or where hotels and malls vs office towers might be.

But it is a lot of data that's very hard to keep updated.

> LeafletJS’s developer is Ukrainian and can’t advance his project

Not sure what you mean by this?

Are you suggesting you can't support the project of a Ukrainian, or that the Ukrainian developer is unable to support their own project due to the war? Keeping in mind that Leaflet is a well established, open source project with any number of maintainers / contributors and (while no doubt an impossible time for a family from Kyiv) @mourner's contribution graph [1] remains remarkably healthy.

[1] https://github.com/mourner

Reply on news.ycombinator.com