Hacker News Reader

Comment by tppiotrowski | Hacker News Reader

One thing I've wondered about are size savings for DEM tiles. Typically elevation values are encoded in RGB values giving a resolution down to fractions of an inch [1]. This seems like overkill. With an elevation range from 0 - 8848 meters (Mt everest), you can use just 2 bytes and get an accuracy down to .2 meters. That seems plenty for many uses. Does anybody know if there's a PNG16 format where you can reduce the file size by only using 2 bytes per pixel, instead of the typical 3-byte RGB or 4-byte RGBA?

[1] https://docs.mapbox.com/data/tilesets/guides/access-elevatio...

View on HN

Not my area of expertise, but in my limited experience DEM tiles are usually GeoTIFF. This can be 16bit greyscale. The catch is that these are actually signed... Elevation doesn't start at 0meters bc you have locations below sea level and you need to handle those corner cases somehow

What's funny is that you can parse a GeoTIFF as a .tiff most of the time but not always. I had fun debugging that :). Java's BufferedImage understandably doesn't directly support negative pixel values haha

Data from sources like GMTED2010 or SRTM15+ is often float32 or even float64: whether it needs to be is another question, but float16 often isn't sufficient in terms of magnitude accuracy (IEEE), and as you mention you often need negative values as well, which for the whole ocean surface of the earth, more than double the range.

To me (working in the VFX industry with EXR being the predominant HDR format), it's interesting that something that compresses a lot better than TIFF (i.e. EXR) hasn't won over in the GIS space, but I believe that's mostly momentum as well as the fact EXR doesn't natively support 64-bit float, but then neither does TIFF really (it's an extension), and the same could be done with EXR (extend the formats it supports).

The PNG format already allows grayscale 16bits / channel images. I regularly use this when rendering out depth maps from blender and ffmpeg seems to handle reading from these PNGs just fine (detecting the gray16 pixel format).

However I don’t know of any DEM tile apis that provide these sorts of PNGs but it sounds like a fun project!

Edit: I found this StackExchange post which shows how to generate 16-bit PNGs with gdal https://gis.stackexchange.com/questions/246934/translating-g...

I hacked up a quick script and converted https://s3.amazonaws.com/elevation-tiles-prod/terrarium/0/0/... to 16-bit (2 channel) PNG.

Any value below sea level I set to 0. Any value above sea level I converted using the following formula:

scaled = elevation * Math.floor[(256 * 256) / (8849 - 0)];

top_byte = scaled / 256;

bottom_byte = scaled % 256;

This provides accuracy to 1/7th of a meter compared to 1/10th of a meter with Mapbox but the tile size went from 104KB -> 25KB. For applications that can ignore elevations below sea level, this is a huge savings.

Edit: the top level tile has a lot of ocean so the size savings are better than average. On a tile with no water, the savings appear to be around 50%.

Not exactly the same, but I did some work a while back retrofitting 8bit pngs as DEMs with non-linear elevation profiles (in the days when Mapbox only supported 8bit dem uploads). This allowed for fine detail in the range I was most interested in and coarser detail elsewhere. I was also working below sea level, so the standard models weren't suitable. I used gdal and node for the encode to PNG (from high res geotiffs) and then leant on mapbox expressions for the custom decode on the front end. Looked cool and file size was reasonable. Tho I'm certain much cooler things are now possible with 16 bit encode and the new terrain API.

Edit: a link to a mapbox-gl-js discussion on the use of custom dems/encodings (after which anything is possible): https://github.com/mapbox/mapbox-gl-js/issues/10775

If file size is your worry and accuracy not, maybe use a lossy format for tiles like JPG or JP2 rather than PNG?

Worth noting DEMs are moving away from tiled formats recently. Mainly to COG (Cloud Optimised Geotiff) which isn't the most efficient but is a simple tweak to a file format already broadly adopted. There's a few others out there aiming for efficency at scale too - ESRI has CRF and MRF for instance, but nothing has become industry standard other than COG yet.

I’d suggest jpeg; lossy compression over the full precision data will get you to your target bit rate trivially.

People get funny about accuracy in maps. Being able to specify accuracy, even a limited accuracy, is worth a lot. Saving bytes by reducing specified accuracy is in a lot of use-cases better than saving bytes by fuzzing data.

I take your point; but I don’t think features on an in-browser map are ever small enough for compression artifacts to ruin the integrity of the map.

You can compute the maximum error when you encode, which tells you how much precision you can still claim to have

How? I thought it was up to the JPEG decoder how to actually decode the image into pixels. (Not that JPEG couldn't be workable in practice if some care was put into a solution.)

Decoding JPEG doesn't leave much room for interpretation and the images should essentially always be decoded the same. For encoding, it's a different story, as there are steps to downsample the image data (chrominance data is often, but not necessarily, sampled at a lower rate than luminance data), there can be a different cutoff point for which of the DCT coefficients to discard (usually related to the "compression" or "quality" setting for the compressor). All JPEG decoders should reproduce the same image from a given JFIF file, but I'd be surprised if different encoders produced the exact same JFIF file from a given source image.

For grayscale data there's very little ambiguity, since chroma and colorspace conversions aren't involved. Basically just rounding in DCT, for which you can make reasonable assumptions.

Moreover the JPEG XT spec (not to be confused with JPEG XL or the ton of other X-somethings JPEG made) has specified precisely how to decode the classic JPEG format, and has blessed libjpeg-turbo as the reference implementation.

JPEG-XL has a lossless mode, and supports custom bit depths and channel counts.

Reply on news.ycombinator.com