Tools I Recommend for Building Geospatial Web Applications | Hacker News Reader

Tools I Recommend for Building Geospatial Web Applications

medium.com

366 pointsDoreenMichele63 comments

Really interesting to see how stable these recommendations are over time. I built a commercial GIS system at a startup almost ten years ago, and GeoServer, PostGIS and OpenLayers were the go-to tools back then too.

I left that startup soon after, but I heard that the backends for the GIS product really didn't have to be upgraded or revamped for years— PostGIS and GeoServer were just running solidly. Really a testament to the developers of those tools.

If you want to use vector maps you should also check out Mapbox GL JS. I use their tiles but can use other providers or generate and host your own, which I also do.

I've been making spatial apps for probably 8 years now and moved from Google maps and Leaflet towards vector maps and I don't think I'd ever go back.

https://github.com/mapbox/awesome-vector-tiles/

Particularly now that it's relatively straightforward to bake and serve your own MVT tiles, so you're not tied into the Mapbox service ecosystem if you don't need any of its other features (and you're happy with the scale / performance of your own tile server(s))

If your data set isn't dynamic you don't even need a tile server. You can generate the tiles with tippecanoe [1] and serve them straight out of S3.

When things get too large you can keep them in the mbtile file and serve them out with an express app.

I've done it in a severless environment which scales well but would get crazy expensive for any high use.

[1] https://github.com/mapbox/tippecanoe

Do you know of any tutorials on how to generate and host own (raster) tiles?

I would suggest starting with something static if you're just starting out. Best way to get a feel for how it works without having to worry about setting up a spatial database.

I used this article myself to learn: https://geovation.github.io/build-your-own-static-vector-til...

There are a couple more articles at the bottom of the awesome vector tiles I linked to before but haven't read them myself.

Reading through the list is good as well because it's a good overview of the whole ecosystem.

Edit: This article also shows a simple way to dynamically serve vector tiles directly from PostGIS. I didn't realise you could do it without the middlewear, time to look into doing it this way!

https://medium.com/nyc-planning-digital/using-the-new-mvt-fu...

Thanks for the explanation on the vector maps! Do you also happen to know how to get started on raster maps, e.g. where to get satellite imagery, prepare and host the tiles?

I am currently interested in it because I want to feed tiles to a GAN to automatically generate maps of an imaginary earth..

Tiling and hosting is the straightforward part, FWIW.

For the reprojection, tile cutting, and overview building in gdal, see https://gdal.org/programs/gdal2tiles.html You can also implement it yourself fairly easily if gdal2tiles.py doesn't have the features you need. The files and directory structure it outputs you can slap into into a static webserver and visualize directly with something like leaflet.

As far as where to get data, that's a whole topic unto itself, but you can start with things like https://earthexplorer.usgs.gov/ for Landsat or https://datagateway.nrcs.usda.gov/ for NAIP or https://scihub.copernicus.eu/ for Sentinel.

I haven't implemented it yet, but [0] looks like a great, detailed tutorial for serving tiles from S3. The article talks about vector tiles, but it should work with raster tiles that you save as .PNG in the same directory structure.

[0]: https://github.com/addresscloud/serverless-tiles

Yes! Particularly OpenMapTiles has a fantastic docker setup to generate vector tiles for extracts or the whole planet: https://openmaptiles.org/docs/generate/generate-openmaptiles...

These are awesome tools. Do any of them support non-earth coordinates? I have a project where I want to do the equivalent of mapping data onto an asteroid (arbitrary shape, including the possibilities of concavities).

What tools would work best / support that workflow?

That has nothing to do with these types of frontend tools and everything to do with projection libraries.

In your case, all of the full libraries like proj4 already support it. What you're describing is a custom ellipsoid and datum. Plug that in, and you're good to go. You'll need to define the shape of your asteroid's equipotential surface through a reference ellipsoid dimensions and a datum that defines deviations from that ellipsoid. Making those is definitely non-trivial, but once you've got the datum grids made, the math is the same as it is for Earth.

Also note that the datum is _not_ the shape of the asteroid directly. It's the 3D shape of a surface that's "flat" gravitationally. In other words, it's the shape of sea level. It's something that depends on the density distribution of the body and it's really not easy to measure, though it is easy to predict if you know the density distribution a-priori.

The shape of the asteroid's surface is topography. That's independent of the coordinate system.

> That has nothing to do with these types of frontend tools and everything to do with projection libraries.

Mapping frameworks need to support map APIs to render stuff, and some APIs such as Open Geospatial Consortium's WMS (OGC WMS) requires clients to specify coordinate systems.

https://www.opengeospatial.org/standards/wms

I'm unclear how that's relevant in this case...

Yes projections need to be specified... The datum and ellipsoid are part of the projection. No, you won't be able to pass your custom datum along to an external service and have it recognize it automatically. That doesn't change the fact that you can define it locally and have your local tools recognize it. proj4 et al will happily let you define custom ellipsoids and register a new datum and work with projections on that ellipsoid and datum. You can view that just fine in QGIS et al.

Also, WMS is pretty niche and not too widely used partly because it supports things like this use case and is therefore quite complex. I wouldn't bring WMS into this. However, if you want to, the projection specifications it uses _also allow for custom ellipsoids_. You can happily request things in a projection with a non Earth ellipsoid -- it's just a matter of whether the WMS server supports it.

>The shape of the asteroid's surface is topography. That's independent of the coordinate system.

I would have stated this first.

You don't really need a equipotential surface or even vertical datum, unless you're trying to model potential energy, uphill and downhill, the flow of a river, or the gravity well. All of which are unlikely to be known or easily measurable on an astroid.

And if all you want to do is look at the geometry of a surface, you don't need, or even want to use a horizontal datum and projection. It will just overcomplicate the math.

Lat long or a similar polar (or elipsiodal) coordinate system using orthometric height from the origin (or an arbitrary "zero" datum height) will be much easier. Especially in the case of a body without existing coordinate systems or projections that need to be "tied-in."

On the other hand, there are cases where using the GPS style coordinate system makes more sense (which is just an x,y,z or x,y,H version of lat, long, height. Yes, at it's heart, GPS uses Cartesian x,y,z). This is especially true if you're not concerned with defining an ellipsoid, prime meridian or equator. (Especially true for shapes that might not rotate, or fit well to an elipsiod or sphere.)

The thing to remember is that coordinate systems and projections mainly exist to translate between the many different ways people have been mapping and measuring the Earth for hundreds or thousands of years. Sometimes there are good reasons for these different methods (shape, area, directional constraints), and sometimes it's advanced tech/ systems (SI vs Imperial, Clarke elipsiod vs Hayford , or convenience/locality (relating Rose line observations with those made in Greenwich), or national security (USA not wanting nuclear icbm targeting stolen by USSR to be useful/meaningful/etc).

I say that to say this: If you're doing everything from scratch, there's no need to complicate things. Stick to the simplest math that works for the projects demands. There's nothing wrong with spherical polar or xyz.

Put another way: don't go looking for a nail just because you have a hammer.

Very true! You don't need a datum unless you're measuring angles on the surface and elevations.

My point was more that the standard mapping tools allow non-Earth bodies just fine. E.g. here's a UTM-like projection for Mars using a simple ellipsoid in proj4: "+proj=tmerc +lat_0=0 +lon_0=0 +k=0.9996 +x_0=0 +y_0=0 +a=3396190 +b=3376200 +units=m +no_defs" You can happily define a simple sphere of the right general size and have things work correctly.

It's not overcomplicating the math to use existing libraries, it's avoiding rewriting a lot of fairly complex operations that are easy to get wrong.

You can use gdal, proj, org, et al quite well on other planets. It's a hell of a lot easier than writing everything from scratch. They're set up to do things like this.

Yes, I didn't mean to imply that the software wouldn't be useful. (Or for anyone to attempt to do all the math on their own.) Most people just don't realize you can define a generic coordinate system, and depending on the software, leave the projections/ datums blank (or set a generic/custom one), and still get good use out of many of the tools. (The danger is some tools that should only be used with projections might run error free without having one defined and return a meaningless answer.

Although admittedly I haven't kept up with the different capabilities for the last few years.

What method/format would even be used to generate points for an astroid? Radar? Lidar? Photogrammetry? From a "where do you start" perspective, knowing that would probably narrow down practical approaches...

You might try JPL's HEALPIX library.

Would be interested to chat some time, I've been doing the same, Uber's H3 has some interesting possibilities for mapping irregular objects.

Could take a look at this recent tool, used operationally for Mars GIS. A development version has some support for Europa: https://github.com/NASA-AMMOS/MMGIS

I’ve used PostGIS with a Flask (and some js) front end when I was working on my MSc. (Mapping deprivation and services by postcode) Worked amazingly well and was relatively easy to get used to once I read the O’Reilly PostGIS book to get used to the function calls. As with any data project; munging the data into a usable format was the biggest hassle.

Edit: used google maps for the display before they changed their licence. Haven’t updated to OSM yet...

Same. I use Flask because it's one of the lightest frameworks I could find. GeoJSON can be generated on the fly by PostGIS with very good performance. So Flask is just used as middleware to submit the query from URL params to PostgreSQL.

For the front-end I usually use Leaflet with Mapbox (raster) tiles. I should switch to vector tiles but "@2x" raster tiles look good enough on HiDPI/Retina displays.

I'm also an heavy QGIS user. I've managed to convince IT to put it in our internal Software Center so users can install it in one click without needing to be admin. It's becoming quite successful internally ;-). Take this Esri!

However we still pay hundreds of k€ to Esri and nothing to QGIS. I've personnally donated a few euros but it's a really small amount compared to what we pay to Esri.

Also worth checking out OmniSci, (https://www.omnisci.com), formerly known as MapD, which can query and visualize tens of billions of geospatial records in sub-second timeframes. See here for an interactive demo of 11.6 billion ship AIS records: https://www.omnisci.com/demos/ships. Note that the backend database is open source, but the rendering engine and web front end are part of the paid offering. [Disclaimer: I work there]

Hi. Checked out the demo. Super interesting!

If I want to become a developer and sell the projects on your platform: How can I start?

What is proximate pricing schema?

Hi John, thanks!

One option is our cloud (pricing is on the page). https://www.omnisci.com/cloud . You can also spin us up from the AWS, Azure, or GCP marketplaces. If you need an on-prem option, feel free to reach out: info@omnisci.com.

Also, I can see that you have ODBC interface for the DB. Any DEMO database I can play with?

If you sign up for a cloud trial, we can provide you with an ODBC interface. Other options are using our JDBC or our Python/JS interfaces, which are in our open source: https://github.com/omnisci .

https://medium.com/@mbostock/command-line-cartography-part-1...

Alt. for java devs:

http://geoserver.org/about/

https://geotools.org/about.html

https://sis.apache.org/

I'm surprised Redis hasn't been mentioned for simple use cases. It has built-in distance and search functionality: https://redislabs.com/redis-best-practices/indexing-patterns...

It's essential if you are pulling any significant amount of data, e.g. shapes for dividing a country into administrative districts will be a couple MB uncompressed pulling from a PostGIS database.

Interesting, looking over the docs I couldn't find anything beyond simple radius queries.

It seems that all redis handles is points.

How would you query from shapes with redis?

Just using it as a response cache for GeoJSON.

There is of course https://github.com/sshuair/awesome-gis

I would be interesting to read more about why a tool was used, also: what are the current limitations?

We started building a web geospatial platform 5 years ago. We migrated from GeoServer to Mapbox VT but still using Leaflet because it brings us a better interactivity on the map.

During the time, we've build a self service GIS SaaS product. In contrast with a layers based GIS tools, it supports a complex Data models as you know from BI tools.

If you are interested, try our free trial: https://www.clevermaps.io/free-trial and documentation: https://clevermaps.atlassian.net/wiki/spaces/CDD/overview

- Which API would you recommend to get accurate transit information data?

- Is there any article talking about and path optimization including time constraint? I've done something homemade (a mix of bird flight distance and time to destination) it looks "okay" but I'm sure can be better.

I've just built a tool to calculate directions for long flights layover [1], but the Google Maps API Directions sometimes lack of those data (Tokyo/Singapore/Istanbul for example). Any recommendations appreciated!

[1] https://travelhustlers.co/layover-in/

I would see if your system has a gtfs feed.

https://developers.google.com/transit/gtfs/

You could use pgrouting for routing.

https://pgrouting.org/

GTFS and GTFS-RT definitely work, but in practice it seems each transit agency implements them a little differently.

I'm actually a fan of the HERE Transit API. They've done the work of interfacing with a bunch of providers, and you can query their whole dataset with a REST API. Unfortunately it's proprietary and only free at small volumes. But I found it much easier to hook up than working directly with a transit agency.

That said, their coverage can be a bit random. For example, I ended up having to do direct GTFS-RT anyway because HERE doesn't seem to have real-time data for NYC subways.

Not a transit routing API, but transit.land offers aggregated transit network data as a service. https://transit.land/

If you are looking for something to handle large amounts of data I would take a look at deck.gl [0]. It useful when you have some rich data you want to layer on top of a high quality map, with built in Mapbox GL integration [1].

0 - https://deck.gl

1 - https://deck.gl/#/examples/core-layers/geojson-layer-polygon...

It isn't clear how this handles large amounts of data? The examples are thousands of polygons, which while useful isn't "large". Drone data models and similar are often trillions of polygons. How do you scale-out the data model processing?

As a tangent, deck.gl (first time I've looked at it) describes their 64-bit floating point computation as having "unparalleled accuracy". It is not possible to correctly compute some double precision geospatial relationships using double precision computation (it requires quad precision or greater to produce a correct double precision result) and other software systems, including in open source, do this correctly.

Everything is relative and in the Web usually large is quite a bit smaller than native, so here I assume “large” really meant “bigger than 2D Canvas or SVG could handle”. This example has a few 100k things [0] so you could judge by how well that works.

Not sure about the 64 bit statement, but I assume it means relative to normal 32 GL shader math. Which is definitely not “unparalleled”, but I assume better than the truncated doubles of native JavaScript or the standard shader math.

0 - https://deck.gl/#/examples/core-layers/geojson-layer-paths

1) I've successfully put >1M points into a deck.gl point layer with cute hover effects, and on a modern computer I can easily get 30fps on a map in the millions of pixels. 1a) Large-scale map viz in the browser is not only challenging for in rendering (see https://github.com/uber/deck.gl/blob/master/docs/developer-g... for details), but it's also challenging for data transfer. If you have a table of 7 variously-sized columns with a million rows of data, you'll probably want to send 7 arrays to the browser, just with the way WebGL is going to unpack parameters to your shaders, so you'll want something like Google's FlatBuffers, and FlatBuffers with named columns and more datatypes is the Apache Arrow format (which has excellent Javascript support, and strives to avoid copying buffers, which is what you want in high-performance applications).

2. By default, many of the deck.gl layers don't use 64-bit floats, they use 32-bit floats for lat/lng degrees, which supports many use cases for resolution and speed. In a WGS84 projection, with a base/meridian of (0°,0°), a point in northeast Russia represented by two IEEE754 32-bit floats (say: 70°N, 179°E) will be within roughly a meter of its original value (as per https://upload.wikimedia.org/wikipedia/commons/3/3f/IEEE754.... and as per the earth circumference is ~40M meters, 1° is at most at the equator ~ 40M/360 meters ~ 100 km, and a rounding error of 10^-5 on a float that's 10^2 means you'll be off by around a meter). The deck.gl layers allow you to provide your zero point, and you get 100x the resolution at 1 compared to 100 in the exponent+fraction format for floats (so pre-render your Δlat, Δlng in 64-bit float computations with standard 80-bit precision somewhere besides WebGL). If you have more than a 100 km x 100 km square for which you need sub-centimeter precision, computed at render-time, 32-bit floats aren't for you. Which brings me to speed: WebGL doesn't have 64-bit floats, or 64-bit ints, natively. You'll have to fudge the math in software in a shader, instead of the native add instructions on the GPU, and that is far slower. If you do need higher precision with only 32-bit integers, you can cluster your points around many different base/meridian points instead of the single (0,0), and take advantage of increased float precision on smaller numbers. Or use 64-bit floating computations with the standard 80-bit precision in Javascript, outside of GPU shaders, and compute millions of floats in Javascript typed arrays to send into deck.gl shaders.

Ive used elasticsearch geoapi in the past. Cant speak to its current status, but it was very helpful! https://www.elastic.co/blog/geo-location-and-search

I have a question that's kind of adjacent to this. I've wanted to play with mapping and especially with geolocating data, but at least right now, I don't want to put much money into it. Are there any reasonably decent tools for doing bulk address to latitude/longitude geolocation, preferably something I could pull down and run in Docker containers on my desktop machine? I'm aware of the Data Science Toolkit, but it seems a bit out of date.

Great article! One thing that wasn’t mentioned was Geocoding — Converting between addresses and coordinates. This is also a very common need for geospatial web applications.

Here’s a couple of options:

geocode.earth -> Worldwide coverage

geocod.io -> US/Canada only, but supports spreadsheets, data appends and batch geocoding

smartystreets.com -> Ideal for USPS address verification

Disclaimer: I’m the co-founder of Geocodio

If you outgrow PostGIS, you may want to look into GeoMesa, which allows you to set up spatio-temporal indexing on top of NoSQL column stores including HBase and Cassandra: https://github.com/locationtech/geomesa

Thanks for the pointer, haven't heard about it. Spatial-temporal indexing sounds interesting!

Do you happen to have any estimates when things start to break down for you in PostGIS? My largest table of PostGIS 3D geometries (line strings) has ~800,000,000 entries and most bounding box queries are still very fast (<100ms).

No, I'm not a PostGIS expert unfortunately. It probably depends a lot on the balance between reads and writes/updates.

I've been using Mapbox for client and personal projects, and so far it's been great but I'd like to give Leaflet a try (for the sake of less dependence on commercial companies for personal non-profit projects), anybody here went down the same road, especially in the context of React applications ?

Leaflet doesn't provide tiles, but lets you get tiles from a variety of sources including OSM, & Mapbox, so you still need a tile provider, or need to render your own tiles.

GeoDjango is really awesome. I’ve used it for a few things geospatial and I had no big troubles with it.

Thanks for the recommendations.

I need to create a polygon of specified width based on an arbitrary GPS track (which can cross itself multiple times). Can anybody recommend a JS library which is capable of doing this?

Perhaps check out TurfJS for client side spatial buffers and point in polygon functions.

https://turfjs.org/

Also leaflet.js and openlayers.js for web mapping.

You should be able to this JSTS I think. Create a line with the gps track, create a buffer with the specified distance, feed the resulting polygon to your mapping library.

https://github.com/bjornharrtell/jsts/blob/master/README.md

Sample use: http://bjornharrtell.github.io/jsts/

For anyone who is looking for good 3D geospatial tools I recommend Cesium JS. It works great for pretty much any amount of data.

cesiumjs.com

(In full disclosure I did work as an Intern for CesiumGS)

In case someone needs geofencing capabilities, have a look at http://www.tile38.com

Suppose one wants a simple world map with one or two levels of zoom and some interactive points plotted over it. What would be the recommended approach?

Can anyone recommend any tools for routing? Ideally with a couple sentence pros/cons breakdown

> Can anyone recommend any tools for routing?

Google's Operations Research Tools has implementations for some vehicle routing algorithms.

https://developers.google.com/optimization/routing

Why not just use geohash and text ranges?

is there a js framework that has most (some) of the functionality of qgis?

For geographic and geometric operations in JavaScript:

https://turfjs.org/ http://bjornharrtell.github.io/jsts/

For projections:

http://proj4js.org/

https://github.com/d3/d3-geo

Reply on news.ycombinator.com