Hacker News Reader

Comment by Game_Ender | Hacker News Reader

If you are looking for something to handle large amounts of data I would take a look at deck.gl [0]. It useful when you have some rich data you want to layer on top of a high quality map, with built in Mapbox GL integration [1].

0 - https://deck.gl

1 - https://deck.gl/#/examples/core-layers/geojson-layer-polygon...

View on HN

It isn't clear how this handles large amounts of data? The examples are thousands of polygons, which while useful isn't "large". Drone data models and similar are often trillions of polygons. How do you scale-out the data model processing?

As a tangent, deck.gl (first time I've looked at it) describes their 64-bit floating point computation as having "unparalleled accuracy". It is not possible to correctly compute some double precision geospatial relationships using double precision computation (it requires quad precision or greater to produce a correct double precision result) and other software systems, including in open source, do this correctly.

Everything is relative and in the Web usually large is quite a bit smaller than native, so here I assume “large” really meant “bigger than 2D Canvas or SVG could handle”. This example has a few 100k things [0] so you could judge by how well that works.

Not sure about the 64 bit statement, but I assume it means relative to normal 32 GL shader math. Which is definitely not “unparalleled”, but I assume better than the truncated doubles of native JavaScript or the standard shader math.

0 - https://deck.gl/#/examples/core-layers/geojson-layer-paths

1) I've successfully put >1M points into a deck.gl point layer with cute hover effects, and on a modern computer I can easily get 30fps on a map in the millions of pixels. 1a) Large-scale map viz in the browser is not only challenging for in rendering (see https://github.com/uber/deck.gl/blob/master/docs/developer-g... for details), but it's also challenging for data transfer. If you have a table of 7 variously-sized columns with a million rows of data, you'll probably want to send 7 arrays to the browser, just with the way WebGL is going to unpack parameters to your shaders, so you'll want something like Google's FlatBuffers, and FlatBuffers with named columns and more datatypes is the Apache Arrow format (which has excellent Javascript support, and strives to avoid copying buffers, which is what you want in high-performance applications).

2. By default, many of the deck.gl layers don't use 64-bit floats, they use 32-bit floats for lat/lng degrees, which supports many use cases for resolution and speed. In a WGS84 projection, with a base/meridian of (0°,0°), a point in northeast Russia represented by two IEEE754 32-bit floats (say: 70°N, 179°E) will be within roughly a meter of its original value (as per https://upload.wikimedia.org/wikipedia/commons/3/3f/IEEE754.... and as per the earth circumference is ~40M meters, 1° is at most at the equator ~ 40M/360 meters ~ 100 km, and a rounding error of 10^-5 on a float that's 10^2 means you'll be off by around a meter). The deck.gl layers allow you to provide your zero point, and you get 100x the resolution at 1 compared to 100 in the exponent+fraction format for floats (so pre-render your Δlat, Δlng in 64-bit float computations with standard 80-bit precision somewhere besides WebGL). If you have more than a 100 km x 100 km square for which you need sub-centimeter precision, computed at render-time, 32-bit floats aren't for you. Which brings me to speed: WebGL doesn't have 64-bit floats, or 64-bit ints, natively. You'll have to fudge the math in software in a shader, instead of the native add instructions on the GPU, and that is far slower. If you do need higher precision with only 32-bit integers, you can cluster your points around many different base/meridian points instead of the single (0,0), and take advantage of increased float precision on smaller numbers. Or use 64-bit floating computations with the standard 80-bit precision in Javascript, outside of GPU shaders, and compute millions of floats in Javascript typed arrays to send into deck.gl shaders.

Reply on news.ycombinator.com