Hacker News Reader

Comment by Soupy | Hacker News Reader

This is ridiculously cool, demo is slick and fast! I work pretty extensively with tiling pipelines and GeoTiffs right now as I'm building out a mass repository and platform for historical map and aerial analysis (https://pastmaps.com - still very early so don't judge it too hard please)

As part of this work, I similarly hit problems with using the raw GeoTiff files as my source and found that I was able to build some custom tiling hooks into MapLibre coupled with http range queries on the static files hosted in S3 to bypass the need for tiling. It does push the compute to the clients but I've found it's actually pretty fast even on older mobile devices.

If this MapLibre GeoTiff source support is at all of interest, I'm happy to share some basic code or even open source some of that work. Here I was thinking I was the only weird dude on the internet messing with this stuff :D

View on HN

Ah. I hadn't even thought that I could do byte offsets on the fly. In general I would err on the side of over-sharing because until ChatGPT came along I struggled to understand what tools and options were to generate tiles as there wasn't much data I could find online.

One thing specific in my case: the LiDAR GeoTiffs are in imperial feet with a 32 bit floating point precision. If you take the elevation range from sea level to Everest in meters (8848) and pack it into an int16, you can get .2 meter precision. That's plenty for ShadeMap so converting from float32 to int16 should save half the cloud storage space in theory and more when taking PNG compression into account.

This code is rough, and that's an understatement, but here's a basic gist showing how I'm doing what I'm doing in case it helps you in any way: https://gist.github.com/craigsc/fdb867f8971ff5b4ae42de4e0d7c...

I'm similarly using R2 as my static hosting backend and it's been pretty fast and seamless

note that i'm using the geotiff.js and fast-png libraries for the heavy-lifting behind the range-queries and the client-side png encoding. why reinvent the wheel, right?

Your browser has a very powerful image decoder built into it, offloading the PNG decoding into Javascript is very resource hungry.

Using maplibre (or any map viewer) you can load blobs of image data out of a tiff and use `Image` or `Canvas` to render the data onto a map.

Its even easier if the tiffs are already Cloud optimized as they perfectly align to a 1-to-1 map tile and they don't need to be rescaled, you can then just render the images onto the map. eg here is a viewer that loads webps out of a 15GB tiff and uses Canvas to render them onto a map [1]

Unless you are trying to layer all your maps together, you also could stop reprojecting them into webmercator, or if your goal is to layer them, then storing them in webmercator would save a ton of user's compute time.

There are a bunch of us that talk web maping and imagery in the #maplibre and #imagery slack channels in OSMUS's slack [2]

[1] https://blayne.chard.com/cogeotiff-web/index.html?view=cog&i...

[2] https://github.com/maplibre/maplibre-gl-js#getting-involved

Great points. Thank you for the links. The one trade off here is that uncompressed blobs will require longer downloads than PNG and I think usually the network transfer is slower than PNG decoding.

But maybe the sample gist takes a Tiff blob and encodes it to a PNG on the client and then maplibre decodes the PNG to canvas. That would be quite inefficient if that's what it's doing.

Those comments were more at pastmaps.

For elevation data, we store our DEM/DSM in S3 as LERC [1] COGS, LERC has a WASM bundle which I think can be used in the browser. We found LERC COGs to be one of the most space efficient ways of storing highresolution DEM/DSM data [2], If you wanted to you could fetch LERC tiles directly out of a remote COG and use that directly for the terrain heights.

I am more focused on storage/archiving/publishing of our LiDAR capture program [3] than web based visualizations of it though, so I am unsure if a LERC COG would even be better for you than a PNG TerrainRGB.

[1] https://www.npmjs.com/package/lerc

[2] https://github.com/linz/elevation/tree/master/docs/tiff-comp...

[3] https://linz.maps.arcgis.com/apps/MapSeries/index.html?appid...

WASM surely could be an improvement over js, especially for kind of BigData-ish/repetitive jobs, and where load on clients might become the next wall after we optimized the cloud/server part, or when we try to use js on cloud leafnodes.

amazing comments and callouts, thank you! I actually tried to load the raw image data blobs as a layer into MapLibre but couldn't figure out a way to do it and finally capitulated and did the "bad" move of re-encoding just to get the initial interactive map collection out the door for folks. It sounds like this is in fact possible but I missed something. I'll take a look at the Image and Canvas sources, thanks!

Re the webmercator reprojection - yea it's gnarly that I'm doing it clientside but it's exactly because I'm working towards the ability to layer them interactively on top of eachother (as well as on various basemaps). My projection code is also only half-working at the moment and it's where I'm currently spending my time next week. I'm trying to avoid building pipelines to re-encode the geotiffs as long as I can since there's 10+TB of them in my backend so this is why you're seeing me doing this clientside instead. This is a solo project so I need to be really picky where I spend my time so I can keep moving the ball forward

I'll join those 2 communities, thank you! Been crazy hard to find folks who are deep in this stuff so most of my learning has been through endless googling down deep dark corners of the web for the past 2 months

> coupled with http range queries on the static files hosted in S3 to bypass the need for tiling

Did you look into Cloud Optimized GeoTIFF format?

https://www.cogeo.org/

It is supported by OpenLayers: https://openlayers.org/en/latest/examples/cog.html

(I don't think that Maplibre or Leaflet have built-in support for it)

I did actually! For my particular source dataset it was far easier just to lean on the plain old GeoTiff format since that's what my source data was already formatted in and my testing of the client-side on-the-fly tiling using range queries and fast-png for encoding resulted in close to par performance with 0 increase in hosting costs and 0 headache in building custom pipelines to encode the cloud-optimized versions. Basically, I'm lazy, I'm sure it's the "correct" answer for other use-cases

I also have been digging pretty extensively into protomaps though for some newer non-GeoTiff datasets I'm in the process of pulling in, in my opinion it's the future for this space - https://protomaps.com/

See also https://geoblaze-gsoc.vercel.app and the underlying libraries it uses, which also do range queries on GeoTIFFs.

I'm super interested in this space, including in helping financially support some projects. I already emailed Ted about this, but would be happy to chat to anyone doing this stuff. My email's in my profile.

(Luckily this is niche enough I'm not worried about my inbox blowing up....)

You're not anymore "the weirdest", You two are unique still though. Share and exchange, and try to support each other. All gonna be good.

I need to learn more where lies the issue with GeoTIFF format. Perhaps "pure conversion" pipeline infrastructure shared between your two projects could help.

If there are two of You, perhaps there are more hitting the same wall.

> This is ridiculously cool

I think so too. And it reminds me that I refactored Leaflet's core event-handling and SVG rendering code 8 years ago. It's good to know your opensource contribution play part in such cool things.

Reply on news.ycombinator.com