Writing a tile server in Python | Hacker News Reader

Writing a tile server in Python

grulic.org.ar

120 pointsaltilunium18 comments

Having used raster MBTiles a lot for offline mapping, I would truly love a generic process that could take vector MBTiles, and then create a fully rendered raster tile from that data with the given stylesheet. If anyone knows of such a cross platform library, I would really like to hear about it. I know of a few projects, but they always have very platform specific code. My use case is multiple platforms, but using the same mapping engine. Something like Skia for rendering and handing back a Skia Canvas/Image with the rendered tile would work very well. We still need "tiles" in other words, but the source would be generated on the fly and cached to reduce overhead,

Raster tiles get huge at level 12 onwards. For generic usage it is hard to take the entire planet and make it available in a way where any user can use the same file at any zoom level in any location. For me, vector would solve this in a much better way, as we have a lot more control over which features to include and what level of detail to have in specific areas to reduce the overall size of the data.

Replying to myself - this needs to be native code really. We are using various .Net UI frameworks, but the mapping is all via Mapsui and Brutile.MbTiles. The current renderer we have is not ideal as it is quite slow and incomplete.

I don't fully understand your requirements, but does mapnik [0] do what you need?

It's cross platform (Linux, OS X, Windows, and BSD) [1] and written in C++. There are bindings for Python/Node/C++.

There is also maplibre-native [2] (which suports Android and iOS as well as Linux/Windows/macOS).

Mapnik is usually used to pre-render raster tiles on a workstation or server. MapLibre Native is a native alternative to the MapLibre GL JS library (both forks of the no-longer open-source MapBox equivalents) for rendering vector tiles in the browser.

[0]: https://mapnik.org/

[1]: https://github.com/mapnik/mapnik/blob/master/INSTALL.md

[2]: https://github.com/maplibre/maplibre-native

Don't want a pre-render, want it to be rendered on the fly on device with no internet connection. Think Android phone or Windows tablet on 2G network at best so bandwidth is not really there.

We basically need a "tilesource" for XYZ PNG based tiles that is fed on the fly from vector tiles and is cross platform enough that it relies only on something like Skia.

This is not what memsom is asking for, but for generating rendered raster MBTiles from any styled source (including vector data), there is this tool that uses maplibre-native: https://github.com/ConservationMetrics/mapgl-tile-renderer

I'm writing something like that in python myself. It's still at an early stage. Sooner or later I'll push it to github even if it's in a crappy stage.

Isn't the simple answer just to use a standard javascript library (maplibre, deckgl, etc.) to render the vector tiles then save that rendered canvas directly as an image?

Not so simple if you want to pre-render and cover multiple zoom levels and a large area

I use MapLibre Native to render static PNGs and JPEGs from vector MBTiles and mbstyle json files.

The nuts and bolts of it boil down to writing a short program in C++ that accepts a few arguments and makes the relevant library calls, and is then called as an external program from my sorbet code. I guess you could PInvoke it but I’ve never got my head around that part of .Net :)

It’s cross-platformish _enough_ for me, as I dev on macos and host on Linux. Can’t think of any reason it wouldn’t compile on windows though.

The flexibility is rather good, and I created a DSL without realising that I was doing so by supplying one program variation with a list of map types, dimensions, starting coordinates + zoom levels etc, which it can iterate over to produce a bunch of static map images.

Highly recommend it, the initial learning curve was steep - for me, at least - but is all one-time learning stuff that you don’t to relearn later on.

The approach I took for a recent project was to generate raster tiles for 0-7 zoom levels on the server. This is done in the data pipeline when new data becomes available. I then load the dataset into memory on the tile server and any requests for zoom 8 and above will generate tiles on the fly and cache them for other requests.

Its worked out very well so far, but I'm working with time series that adds a new set of data every 5 minutes so I need the "on the fly" approach. After some heavy optimization it's become quite fast and often the generated tiles are returned before the base map layer is.

The upside is that it works with any map system that supports raster tiles, so any platform really.

This like protomaps? https://protomaps.com/

hmmm, no. PMTiles are a on-disk format for storing data, in particular, raster or vectorial tiles. My tool is about rendering such tiles on the fly, in particular raster tiles. I chose a different on disk format to speed up the sync'ing between my rendering machine and the serving machine.

>But Flask does not give control of what happens when the client suddenly closes the connection.

It is so annoying that they don't give you a way to do this. I built an LLM inference API but am still working on how to cancel an inference job if a client cancels. I really don't want to rewrite my entire server in Twisted so I'll probably end up just building the HTTP responses like you did.

Flask is FOSS, maybe it is not too difficult to add support for this yourselves.

I believe the WSGI spec is a bit unclear on this but some WSGI servers can give back control to the app when the connection gets closed. I use Flask and Waitress for long-lived server side events and my app is notified when the client closes the connection before the end of the stream of events.

Can you not use teardown_request to run some clean up code after each request?

Note that titiler[0] exists, which is doing a pretty nice job.

[0] https://github.com/developmentseed/titiler

Cool, I didn't know it, although TBH it's quite more generic to what I did, so I think it's even a better tool.

In any case, it was fun to think about this problem and learn things about the limitations of some of the async frameworks.

Reply on news.ycombinator.com