Show HN: Flatmap – a new tool to make vector tiles from OpenStreetMap data fast | Hacker News Reader

Show HN: Flatmap – a new tool to make vector tiles from OpenStreetMap data fast

github.com

158 pointszylepe28 comments

Creator here. I maintain https://onthegomap.com as a side project. I originally used Google Maps API but after their price hike switched to OpenStreetMap data. I self-host GraphHopper for routing and tried to self-host OpenMapTiles but gave up and used Stadia Maps since it would have taken over 100 days [1] to generate a map of the world myself. Stadia Maps has been great and I highly recommend them, but the problem continued to nag me, so earlier this year I started prototyping a new way to generate vector tiles faster. The idea worked and ended up being able to generate a ~100GB planet.mbtiles file in as little as 59 minutes on a single c5ad.16xlarge instance with 64 cpus and 128GB RAM (although smaller machines appear to be more economical).

I’m open-sourcing Flatmap today. Let me know what you think, or if you have any suggestions!

[1] https://github.com/openmaptiles/openmaptiles/issues/654#issu...

I use onthegomap almost daily. Such a great tool. Thank you for it.

Thanks! Glad you find it useful!

Looks great! Potentially a game changer for projects that need a basemap for a smallish area (like one country), get a fair bit of traffic, but don't have budget for ongoing tile subscriptions to Mapbox or MapTiler.

This is really sweet, awesome work. I really love onthegomap too.

I maintain an openmaptiles pipeline, and we've got our planet runtimes down to about ~4 days on AWS.

Would be excited to see if we could use this tool, but we really heavily depend on PostGIS data manipulation to ensure that the data we want ends up in our output tiles. How difficult do you think migrating this SQL to your tool would be? It seems like you can support modifying the data fairly heavily, both before and after MBtile creation.

Flatmap calls into user-defined profiles in 2 places: first when processing each input element to map it to a vector tile feature, then a second time right before emitting all vector features in a layer.

That second call lets you manipulate vector features on each tile using JTS geometry utilities (i.e. merge nearby polygons or lines with the same tags). PostGIS uses GEOS which is ported from JTS so you have access to pretty much the same geometry utilities - often with the same name.

It's definitely not as flexible as a PostGIS based solution - especially if you join faraway features that don't appear on the same tile, but in practice it was enough to port the entire OpenMapTiles schema.

See the basemap layers package: https://github.com/onthegomap/flatmap/tree/main/flatmap-base... for example see the landcover layer https://github.com/onthegomap/flatmap/blob/main/flatmap-base... which was ported from the SQL contained in https://github.com/openmaptiles/openmaptiles/tree/master/lay...

Also, this OpenMapTiles PR might help improve your existing pipeline after it gets merged: https://github.com/openmaptiles/openmaptiles-tools/pull/383

Edit: formatting

Nice :)

There's an ongoing effort to research vector tile solutions for openstreetmap.org. I wonder how your software stack compares to tilemaker, which has seen some optimization lately.

https://github.com/openstreetmap/operations/issues/565

Thanks! Tilemaker is a great tool and probably the closest alternative (no intermediate db). The most recent benchmarks I saw though were around a day for the planet, so roughly 10x longer than flatmap. If

Edit: benchmark link https://github.com/systemed/tilemaker/issues/315#issue-99432...

Nice tool OP, but I think we as a community should try to avoid creating tools that use already common names.

Googling "flatmap" gives a bunch of bad results. There is a value in having unique names.

Hear me out: "mapFlat"

This looks great!

I hacked around with OpenMapTiles to generate vector tiles in Arctic, Antarctic and WGS84 projections (the latter with 2×1 tiles at zoom 0) [1]. The time spent waiting for PostGIS to do reprojections etc made development very slow, and I've put off updating the tiles.

This tool is reprojects from WGS84 to Web Mercator while reading the tiles, so optimistically changing that would be much of what's needed.

[1] See the map at the bottom of gbif.org, there's a button to change the projection

Thanks so much for this tool! Looks like one could generate a planet with less than $5 on Digital Ocean.

I generated the planet a few times in testing:

with z13 building merge (extra 14 CPU hours of processing):

  - $4.68 on Digital Ocean 128GB 16 CPU instance (4 hours @ $1.17/hr)

without z13 building merge:

  - $2.75 on AWS EC2 c5ad.16xlarge instance (1 hour @ $2.75/hr) 
  - $2.88 on Linode 128GB 50 dedicated CPU instance (2 hours @ $1.44/hr)

Unfortunately it costs $10 in egress bandwidth fees to get the result out of AWS ;-)

Edit: formatting

A bit off-topic, but do people who host their own map tiles worry that other apps will use/abuse their tileserver? I would worry that I'd come in one Monday morning and discover that I've got a $20K bandwidth bill because someone else got popular.

Nothing is foolproof, but for web-based traffic, proper CORS configuration or rejecting traffic based on the referrer (or lack thereof) would go a long way.

We self-host OpenMapTiles, but are super lazy about replacing them with updated tiles (and we don’t love the watermark). I’m really glad that there are options for generating them ourselves! Starred, and will give it a try soon.

at that cost it almost feels like something internet archive or aws open data could provide on a regular basis. the cost of pre-made vector tiles is huge, and making them yourself has been a huge ordeal before this...

> at least 1.5x as much free RAM as the input .osm.pbf file size

Ouch! Is there any plan to reduce the amount of ram or it's impossible?

I'm currently building a tile server with Europe's map, the PBF is 28GB, I have 32GB ram.

The benchmark says the whole world can be done on a "DO 16cpu 128GB" machine in 3 hours, so about $3 from Digital Ocean. Call it $5 to allow some time for transferring data.

Just rent the VM.

You're right, at that price it's a no brainer.

The main driver of memory usage (~1.0x the input file size) is storing node locations so we can convert the list of node IDs on each way into lat/lon coordinates. By default nodes are stored in a memory-mapped file so it can run with less than 1.5x the input file size, just slower because of all the page faults. Try running with -Xmx=16g to leave 16GB free for memory-mapped file. it will print % complete as it works through the ways in pass2 so you can decide whether it's worth it to keep waiting.

Also you could try switching from the default --nodemap-type=sortedtable to --nodemap-type=sparsearray. Sorted table uses 12 bytes per node and is more compact for extracts, sparse array uses 8 bytes per node but wastes some storage when there are gaps in the ID space.

There is also an osm.pbf extension that embeds node locations in ways, but you need just as much RAM to convert a file to that format: https://docs.osmcode.org/osmium/latest/osmium-add-locations-...

Once you generate the mbtiles file, you can serve from a much smaller machine.

Thanks for this detailed answer.

This is cool. I will dig in deeper and take a look. I'm working on a spherical display.. so maps with fast zoom will be a must.

Fantastic work! Are the vector tiles compatible with OpenMapTiles styles?

Thanks! The default "basemap" style is ported from OpenMapTiles v3.12.2.

That's great to hear. I'd love to create some vector tiles with building and building height data on zoom levels down to 11. Is this the correct place to perform this change? https://github.com/onthegomap/flatmap/blob/e402a3333bc5127e5...

Or could I even achieve this via the config file? https://github.com/onthegomap/flatmap/blob/main/config-examp...

Yep, that first one will do it! And you probably want to rip out the polygon merging code and minimum zoom on the attrs. You can use --only-layers= or --exclude-layers= if you don't want all the others.

Clever name... but not exactly unique.

Reply on news.ycombinator.com