A new way to make maps with OpenStreetMap
protomaps.com
protomaps.com
I spent part of today on the OSM wiki trying to wrap my head around Mapnik, how tiles are generated, how I am going to host this cheaply because my project is non-commercial, and then the HN gods smiled on me. Thank you for taking this on and I'm coming along for this journey!
One of the main challenges when entering a highly specialised field is finding the vocabulary to express yourself.
[0]: https://openlayers.org/en/latest/examples/mapbox-layer.html
So long-term I am looking into generating and hosting my own map tiles.
To elabore on my comment, I was more wondering about how I generate those tiles from non-tiled raw PBF country extracts on Geofabrik.
Also check out https://github.com/kothic/kothic-js, an older but similar attempt at making a full-fledged map renderer using 2D Canvas. It has a pretty good label rendering in particular which you might find useful.
I am super excited about the parent tool, but unsure whether/when I can generate these bundles programmatically. I tried the web based tool and it works great, but what if I need to do it many, many times?
Can your tool do this easily? Is there a way to generate tiles (tiles.pmtiles) that are imported in the same way?
If I had stumbled on your project, I would not have been enthusiastic about it with just the writeup as-is. Seeing the parent project makes me see the possibilities: a completely independent mapping solution, free of any server/service tie in and additional costs. That is super exciting, and I would love to know if and how your project compares.
The biggest downside thus far is a good way of displaying the data. mtb.waymarkedtrails.org gets close, but I'd really like something that displays relations in the specified color, has sane intersection markers, etc.
I feel like there should be something out there which effectively lets me design a style sheet and then apply it to a map which I embed somewhere (maybe even in an app).
This... feels like it? Or if not this, can any of you recommend another system?
Ideally I'd love something I can self-host. I tried getting a copy of waymarkedtrails going and then modifying that, but it's a bit too specialized and I kept running into quirks getting the toolchain set up.
EDIT: Looks like this might not include the highway=path data that I need, much relations. Alas.
The developers are fairly consistently adding new content as well, might not hurt to reach out.
The end goal is to build a system that lets you customize the tileset to include any node, way, relation, or tag in OSM. This gets quite tricky when considering automated cartographic generalization (how such features will appear when zoomed out) but it's one of my next steps.
As an example, take a look at this: https://mtb.waymarkedtrails.org/#route?id=6109588
That's kinda what I want but... better. My OSM username is the same as here if you'd like to see my contributions; I've mapped most of these systems.
A groomed snowmobile trail addition would also be great!
I am developing another way to run your own map server (without mapnik!) from a low-end computer (https://github.com/jamesrr39/ownmap-app in case you're interested), but a little bit different to you in that I'm currently focusing on rendering raster tiles. I'm really happy I saw this thread, however, I hadn't heard of the OSM Express, MBTiles or PMTiles formats before and was/am rolling my own format, so it's really interesting looking at some ideas from these.
Thanks again for sharing your project!
It's certainly the case that I just haven't built enough mapping-related software to be comfortable but if there's a resource out there that's like "baby's first GIS application" I'd love to read through it and try to ram the concepts into my gray matter again for the day when I do work on one.
> Created by Steve Coast in the UK in 2004[0]
Thanks for starting this way back then, the amount of value created directly and indirectly is immeasurable.
That said, I'm very doubtful of the likelihood of me contributing something novel and useful to the GIS world -- I'm keep to learn from and stand on the shoulders of giants that got us here for at least the first bit of my foray into GIS before trying to contribute something fresh, RTFM and all that. To be fair it has taken 15 years to whittle down this particular wheel, starting from scratch feels almost disrespectful -- I'll leave that for younger explorers.
http://leafletjs.com https://shapely.readthedocs.io/ https://github.com/libgeos/geos https://rasterio.readthedocs.io/ https://gdal.org http://www.angusj.com/delphi/clipper.php
One of the nice out-of-the-box systems I've come across is Pelias[0], what are your opinions on it? It bundles and provides the API so it's one of the things I've thought to self-host if I'm going to. It looks like it doesn't do tiles so I'd need to find something for that?
From deep in my bookmarks I also have:
- https://github.com/tidwall/tile38 (in-memory search+geofencing+other)
- https://openlayers.org/ (visualization, alternative to leaflet?)
- https://download.geofabrik.de/ (where to actually download the OSM data, to hand it to something like PostGIS)
- https://github.com/maptiler/tileserver-gl (A tile server... which presumably needs OSM data & takes coordinates with bounding box parameters?)
> I do have a decade of experience building web mapping applications > I don't have a GIS background in particular;
It's only on HN that you get statements like this, haha, what do you think 10 years of experience building mapping applications is? I building mapping applications is generally what people consider "GIS background" -- if you did GIS but never produced any maps... would the trees make a sound?
[0]: https://pelias.io/
* The technical part of the academic Geography field
* Data collection and mapping work done at agencies like the USGS and municipal governments
* Work done with ESRI software (see https://joemorrison.medium.com/esri-cant-be-stopped-3b063f3d...)
Yes, Pelias is a mature project backed by a great team at https://geocode.earth - definitely one of the best options if you need an autocompleting geocoder based on OSM+other data like http://whosonfirst.org
About OSM data extracts, I also run an extracts service at http://protomaps.com/extracts that provides on-demand data updated once a minute.
Some basics:
Software-wise, GIS is a database hooked to mapping software so you can query the database and spit out maps as your answer. It's valuable because "A picture's worth a thousand words" and most data can be connected to a mappable bit of data (address, lat-long, etc).
A Geographic Information System has the following components:
1. Hardware
2. Software
3. People
4. Procedures
5. Data
6. Network (added more recently and not part of what I was taught, https://learn.canvas.net/courses/464/pages/unit-1-dot-2-iden...)
Data is the most expensive part of that. Good data is hard to come by.
People with the right skills and talents are the most critical part.
You can start a GIS project by surveying your in-house software, hardware, data flows, etc and putting together a use case. You probably already have some of the pieces you need to create a proper Geographic Information System where the data you already track can be hooked into a system for making it query-able and mappable so you can up your decision-making game.
ESRI is probably still the world leader in GIS software and they put out a certain amount of free info.
If I want to build a free open-source web application with a map view that can be annotated, works on mobile etc and uses Open Street Map data, what are the best options right now?
I believe _with some trickery_ you can use Google Earth for that
That said there is a company (nextburb.com) who's CEO I've been in regular contact with who is trying to dramatically scale what kind of searches people can do. I've been talking to them about our needs and use-cases and they've been quite responsive.
Is that comical, or is there a deeper meaning I don't understand?
A frontend developer who wants to put a map in their application and wants control over the hosting and data faces some obstacles when using MBtiles:
* Need a virtual machine and disk space
* Need to choose a particular program to read the MBTiles and serve it via HTTP
* Might need reverse proxy and service monitor (systemd or sysv init) to keep it running 24/7
* Need to deal with SSL termination
The popularity of serverless platforms = evidence there's a strong market for delivering software without all of this.
So that you can FTP some html, js, css, images and map data files to a web host and call it a day.
Source: I created MBTiles ;-)
Deduplication is also important to save space on redundant ocean/earth tiles; MBTiles can deduplicate via views, PMTiles deduplicates by reference, but unpacked Z/X/Y tiles on S3 can't.
As others pointed out, the MVTs inside can be unpacked for static serving. But a SQLite database is not itself usable in this way.
Currently complex tools such as QGIS and GDAL do everything, but the learning curve is very steep.
An addon you might consider a feature to drop an image (and adjust it to fit) over the OSM data to generate a XYZ tilesets directly on the customer AWS S3 (or equivalent storage).
This might be a good way to monetise your offering... I know my customers would buy it.
Curiously, the editor does not render (well, it renders a black screen) in Firefox 78.9.0. I see this error in devtools:
Uncaught SyntaxError: expected expression, got '=' protomaps.min.js:1:52669
On props update af913d4d.0fa8ea22.js:1:7975 ReferenceError: protomaps is not defined
Do you mean it when you say on the blog that OSMX is faster than a relational database? e.g. faster than PostGIS.
And I presume you mean because of the up front cost of conversion?
I personally avoid blanket statements about a certain design being faster than relational databases; for this project as a whole I try to take a "systems programming" approach; I make things faster by doing less work, in this case because I am designing for one specific use case and no more.
For OSMX vs. PostGIS, this entails:
1. PostGIS is a client-server database, OSMX is embedded (in-process). PostGIS must be written defensively for cases like multiple writers; this isn't important for a system that occasionally has a single writer.
2. PostGIS indexes must be built for arbitrary query plans, OSMX only has two kinds of access patterns: by primary key and by S2 cell. S2 cell indexing is an approximation, but will always return more data than you asked for and never less. This is an acceptable downside.
3. PostGIS must treat geodata as OGC Simple Features like points, lines and polygons, this involves lossy conversion from the OSM data model, which is topological.
4. OSMX is built on LMDB, which is an outstanding and underrated piece of database tech (I think the author posts on HN). It's paired with a message format that does not require deserialization (CapnProto) so certain operations common in working with OSM data, such as accessing the coordinates for an OSM node, do not need to malloc for each node. OSM data can be many times larger than RAM; memory-mapping the entire database means you don't need to write your own caching layer; spatial access patterns exhibit high locality since cells are arranged on space-filling curves.
I have tinkered with PostGIS before, and it seemed great. Nonetheless, you make several good points about how a special purpose tool/format can be faster by avoiding certain work.
As for LMDB, I haven't used it, but I have heard about it previously. A full-text search server called MeiliSearch uses it internally.
---
Separately, I think your broader project is really cool. Simplifying the whole end to end process is incredibly useful. I have previously thought "I would love to host a regional map, but I have no idea how".
Can you email me at brandon@protomaps.com with what specific tags and relations you are interested in? I want to support transit layers as part of basemaps and ideally this is an option when generating a PMTiles archive.
I've been using leaflet for my map tiling and it just works. The advantage of promomaps being offline is great.
So, now... how does one add their own POI's, routes, polygons??
1) If your information belongs in OpenStreetMap, you can add it via an editor like the web editor at https://openstreetmap.org - this will also benefit all other OSM users. You can then "refresh" your Protomaps download to get a new map.
2) If there isn't many point and polygons, it may sense to add them as Leaflet layers, especially if you want them to be interactive
3) Other options are creating vector tiles of your own data and merging or displaying them in the renderer (https://github.com/mapbox/tippecanoe is a great tool to do this from GeoJSON) but I don't have much to support this yet.
I'm getting old LOL... my question re adding POIs, etc. was a brain-fart. As you note, I'll just use leaflet layers!!
Been playing around a bit with the protomap's pmtiles.
I notice that POIs have bbox, geom, and just a couple addt'l properties (e.g. name, amenity).
Would it be possible to have the option of exporting the street address into the properties of each poi?
I'd like to be able to (fully offline) generate a delivery route given a list of addresses (that fall within the specific pmtiles region).
Any suggestions or possibility of including this data?
The reason I'm doing it this way is because data completeness and performance - the total size of the tiles - are direct tradeoffs. If I keep augmenting the dataset with tags people request, they'll become too big for everyone!
So I'm exploring this idea of "a la carte" tags in tile sets, but am still gathering requirements. Can you email me at brandon@protomaps.com with more details of the OSM tags you want, as well as the area you're interested in so we can work together on checking out the data availability there?
Not really sure what problem this is trying to solve.
(posted something similar to another response)
The problem with this is it still requires a running server process colocated with that sqlite db in order to service requests for individual tiles, like what you'll receive from a client-side mapping library.
This project has a lot of different parts, but one of them is a spec for serving this type of data at low latency without a server by combining a custom packing format with S3 range-get requests to read individual tiles direct from blob storage. So that is certainly interesting for this kind of use-case.
> coming soon...
> "In about five minutes, you can select any area in the world and get a self-contained map that runs locally, offline or serverlessly on S3 - check out the Getting Started guide."
5 mins? It took me a night to build the map system using openstreetmap in the original way