Replacing Mapbox with open source solutions
kschaul.com
kschaul.com
I'm involved in many of these projects (helped get MapLibre started, wrote a library supporting PMTiles, have used OpenMapTiles for years), and to see open-source, open-data win in the end is really satisfying.
If anyone has any questions about how you could do this for yourself, I'm happy to answer! (I have a company offering SaaS maps, but we know SaaS isn't for everyone, and are happy to point people the right direction when self-hosting is what you want.)
A pivot at some point to these types of archives is an option, but I'd like to roll forward existing customers as well.
Feel free to open an issue.
I'll take a look to see if MBTiles will get me there and if not I'll come back to these.
Thanks heaps
We burn through MapTiler free tiles on a regular basis and have been looking to migrate to self-hosting, but somehow PMTiles flew under my radar and every other approach seemed to either still rely on MapTiler for data (where you have to chose between outdated or expensive) or just be a huge pain to set up. Hopefully this finally solves our map issues so we can actually start promoting the project without fear of the map disappearing in the first week every month.
We have plenty of server resources from both our local partners and various cloud providers that have non-profit grants, but basically no regular budget to spend on SaaS, so self-hosting everything is the only thing that makes sense.
I'd take a look at https://github.com/onthegomap/planetiler -- I bet that fits the bill for what you need if you have the servers but need the data. :)
This is the crux of this. Let’s not pretend that much of this is anything other than “open source clones of Mapbox”.
PMTiles is genuinely innovative and I’m delighted to see it getting traction.
Maplibre GL (which I use and like very much) is a fork of the last open source version of Mapbox GL, with not many code changes between now and then.
Maputnik is an open source reimplementation of Mapbox Studio. Mapbox went on to hire its original dev.
OpenMapTiles started life as basically a how-to/makefile pulling together the disparate, poorly documented parts of Mapbox’s open source tile generation code. It’s now mostly known for a reasonable, not particularly exceptional vector tile schema.
It’s great that people are now shipping production vector tiles without paying the Mapbox dollar. But, PMTiles aside, this is still Mapbox’s world. The next challenge is to evolve the tech stack to something beyond what Mapbox worked up five/ten years ago.
Isn't this a bit like saying we're still in Google's world because most maps still use Web Mercator? :)
Good tech builds on what came before. Mapbox did a lot of ground-breaking work in building tooling around OSM, but so have many others. The fact that they named it Mapbox Vector Tiles is genius in hindsight, because even though we may use tons of tooling they didn't create to build and render them, their name is still there.
> The next challenge is to evolve the tech stack to something beyond what Mapbox worked up five/ten years ago.
Agreed, and I think we've seen a lot of iterative work in the open since then. The next challenge is likely building a OSS stack to do proper 3D: open data (including OSM) to pixels, and that work is already beginning across a lot of organizations: https://github.com/nyurik/future-mvt/discussions, Overture Maps, MapLibre, etc.
DeckGL for instance is a very innovative js library for front end (really developed by Uber I believe), with better 3d and movement, better with dealing data at scale, etc.
Prettymaps is a pure python approach to rendering OSM data.
Etc.
Five years ago Google started to charge an outrageous amount > $1000/month and we could not continue so we switched to something that cost $100/month. And that was for only the maps. We had to scrap the distance matrix and address autocomplete.
Today it's still the same status, and if you want something cheaper then you have to work on putting pieces together like the Washington Post does.
All that functionality for a lot less than $1000 / month.
No idea which customer you are :), but if you ever want to have a chat or have feedback, ping me!
This site made a little bit of money, but not that much. But then 5 years ago google raised their prices so much that we were suddenly in the red. We had to drop the places API (replaced with Yelp) and the autocomplete API (we don't even use that anymore). Recently we've been moving to Apple Maps API which has a pretty generous free tier. Stadia Maps also looks interesting.
Interesting name, considering Stadia was the brand of a recently-discontinued cloud gaming platform made by… Google.
Brandon is a solo-developer who builds Protomaps alone and we get all his great stuff for free...
(Disclaimer: I'm on the governing board, but derive no financial benefit from it.)
"OpenStreetMap is the largest open geographic database in the world, the data infrastructure for multitudes of mapping projects around the globe. Your donation to the OpenStreetMap Foundation will cover our core operational expenses in supporting the OpenStreetMap project: hardware costs, legal fees, administrative assistant and other expenses of our working groups and administration.
We currently run extremely lean for an operation for a project the size and importance of OpenStreetMap. The OpenStreetMap Foundation relies on revenue from individual and corporate membership dues, profits generated by the annual State of the Map conferences, and donations. We must keep our income sources diversified, as these vary from year to year, but our modest needs stay the same. For this, we need your support."
this is amazing to see! we are happy to see our OS contributions like tippecanoe are being used by people all over the place, and i personally really love to work with journalists.
happy to answer any questions!
Going to take a look at replacing them with PMTiles and my own polyline rendering.
Will you be talking about this work openly somewhere?
Here's one alternative, if you don't want to build from scratch: https://docs.stadiamaps.com/static-maps/ We don't have much in the way of query string limits, so have at it!
How are you using the Static Maps API currently where you would be able to send a POST request and display the image back to the user? Most use-cases for the Static Maps API involve the HTML <img> tag or something similar, which all send GET requests.
We wanted to do things even faster so we started migrating to ECS which when using a network optimized instance we could render a full page map in sub 100ms.
The author states that having all the data in a single file for PMTiles is advantageous, but it is a tradeoff. First of all you cannot modify parts of a S3 file so if you want to modify/update your dataset you need to re-process and re-upload the entire dataset. This can be a real pain if you are working with many TB datasets and you just want to update a subset of the data. Also, I thought that there is a global throughput limit on S3 files so wouldn’t putting all data in a single S3 file potentially hit bottlenecks?
There is a throughput limit on S3 files of approximately 5500 GETs/sec per key. Bare archives on S3 is an appropriate choice for small-scale, zero maintenance deployments. If your application demands any thing close to that level of throughput, you're probably either:
* Serving individual tiles over the internet: you should use the CDN integration http://protomaps.com/docs/cdn ; most tile requests will be cached and only misses will interact with the S3 bottleneck.
* Bulk accessing a spatial subset of tiles: You shouldn't be requesting HTTP GETs for single tiles, but instead entire subsets of tiles with a single Range request made possible by the internal Hilbert curve ordering. This is still WIP here: https://github.com/protomaps/go-pmtiles/issues/31
Since Lambda presumably has a separate GDAL RAM cache for each request, I'm guessing this wasn't able to take advantage of any of GDAL's header or block caching? It's pretty impressive you were able to get latencies that low with that approach.
Also check:
https://github.com/protomaps/PMTiles
https://github.com/protomaps/PMTiles/blob/main/js/examples/m...
For the non-tiled parts of the PMTiles file, like metadata and the directories that store Z/X/Y to offset information, those are LRU cached by the PMTiles implementation. The most widely deployed right now is for TypeScript/Node. The "client" can either be web browser or inside a runtime like Lambda/Cloudflare Workers. On those the cache will persist across invocations using the ephemeral memory of the serverless function.
A level deeper, browsers vary in how they cache Range requests; IIRC Firefox does not cache while others will treat Range as if it was any other HTTP header.
I actually used to work with Nokia and Nokia Maps (in Berlin), which is now Here Maps. So, I know a thing or two about this market and still know some people working with various map companies, including at mapbox.
Mapbox is increasingly an alternative to Here, Tom Tom and a few others that offer a wide range of services. The most simple one of which is showing maps. Think of indoor maps, traffic services, routing, geo coding, address search (surprisingly hard problem with open data), etc. Customers for these services tend to spend a lot of money on getting all sorts of bespoke solutions. Maps are a commodity at this point and there's very little money in providing just some basic maps. It's all the other stuff that's built on top of that that is a lot harder and more valuable.
The recently announced overture collaboration could actually shake things up a little in this space. Openstreetmap is nice but also has some pretty significant limitations. Basically what the above mentioned companies do is creating proprietary services and data (with and without open data typically) that they then sell access to for a lot of money. The big companies behind overture are each big customers for this stuff. And now they are pooling resources to create a cheaper, more open alternative to all that.
It's not perfect, and you don't see the full benefit of a WebGL renderer, but if you want to keep using a Leaflet API, it's great.
The trickiest of these assets is the stylesheet, necessary for the renderer to know how to draw the data in tiles. The article mentions Maputnik, which works great for quick prototyping and the production deadlines of newsrooms, but limiting once your map, application, and or tiles get to a certain level of complexity.
Geoserver does fill a pretty big hole in the capabilities space -- it's pretty easy to get going with a bunch of layers and style them, but ultimately they're implementing a RDBMS in xml files, and it's a big, complicated, java system that's been one of the more troublesome portions of the stack (IME).
The other thing that annoys me is that they've more or less abandoned all their open source repos without even as much of a notice in the README. All these amazing tools sitting there with open pull requests and no one responding for years. If they just did a clean break, put a notice on everything they weren't going to touch again, it would have been a lot better/easier for the community to fork and move on with their lives.
I wonder what has changed?
The hosting and bandwidth costs are huge, and most of your customers (by raw count and nominal usage) are hobbyists or free products unwilling to pay even a few cents per user.
Hosting and bandwidth are key inputs to your costs, but they are manageable.
It's gone poorly since then though. SoftBank SPAC hunts new merger partner as Mapbox deal falters: https://news.sky.com/story/softbank-spac-hunts-new-merger-pa...
I wonder if Strava's recent acquisition of fatmap.com eventually boils down to them cutting ties with mapbox via acqui-hiring some mb-libre excellence. I could imagine Strava, due to the extreme map-centric nature of their product, bleeding quite considerable money to mapbox.
Download a region file from maptiler, or make one.
That's it.
It's about 2h work, using certbot for certificates.
If you want to create your own styles, it's slightly more fiddly, but essentially it's https://maputnik.github.io/editor/#0.41/0/0
Host on hetzner for €3/month.
1. https://openmaptiles.org/docs/ 2. https://openmaptiles.org/docs/generate/generate-openmaptiles...
Then mapboxgl or libremapjs on the frontend, or even OpenLayers.
OSM.pbf file -> ogr2ogr -> PostGIS?
I currently use a .mbtiles file and plug it into tileserver-gl, but i guess in theory you could just import OSM to PostGIS and then add the PostGIS as a source in tileserver-gl? That would be amazing
I don't know what they use, but it doesn't seem there is any software that is simple enough to use.
I just want to configure a subregion, call generate-tiles.py planet.xml or whatever, and be done with it.
I like open source, but there will always be open source software that is needlessly complicated because, well, it lets companies like mapbox and others to generate income for their expertise.
That's not the best open source spirit, in my view.
I assure you, the OSS (and in this case, OSM) community will steadily build out better, faster software that includes more and more of the features a modern map needs. It's already looking like companies like WashPo are seeing the potential here.
Come join us and make the world a better place :)
Super easy way to generate a MBTiles, which you can then serve directly, or further convert to PMTiles, which can be used to host vector tiles for client-side rendering using MapLibre (or other renderers).
Raster tiles are a lot harder because you have to generate them on the server, and that's a lot more resource intensive.
They do have more details on the Docker container approach in one of the pages as well: https://switch2osm.org/serving-tiles/using-a-docker-containe...
Here's the relevant Docker Hub link: https://hub.docker.com/r/overv/openstreetmap-tile-server
The requirements are considerable, though, given the workload:
> Serving your own maps is a fairly intensive task. Depending on the size of the area you’re interested in serving and the traffic you expect the system requirements will vary. In general, requirements will range from 10-20GB of storage, 4GB of memory, and a modern dual-core processor for a city-sized region to 1TB of fast storage, 24GB of memory, and a quad-core processor for the entire planet.
Personally, for lower end mobile devices raster tiles still seem to perform better, when compared with at least some of the current vector tile implementations. I actually recorded a short comparison on my phone a while back: https://www.youtube.com/watch?v=A-IRtBGO9rM