Comment by perrygeo | Hacker News Reader

Comment by perrygeo | Hacker News Reader

The problem with a self-described "problem-based guide" to coordinate systems is that it relies on the user to identify the problem in the first place.

Reality is that these are expensive errors that can be avoided by systematic planning but made far too easy by default software options. I've seen weeks/hundreds of thousands of dollars wasted by proceeding with data analysis without properly considering the coordinate reference system.

If spatial accuracy is a concern and

- your data is vector-based lines or a polygons, there is no guarantee that the edges between your vertices will follow the same trajectory in a different coordinate system. This means that the spatial relationship (intersection, overlap, etc) between two geographic objects depends on the coordinate system and the relative density of points.

- your data is raster-based, you need to resample pixels values to translate between coordinate systems. If your pixels represent absolute/measured values (eg population), you need to consider this carefully.

- In both vector and raster, you need to deal with dateline and pole issues. The earth is not a cartesian plane.

- In both cases, you also need to consider more than the projection. You also deal with datums which define the relationship between the actual earth surface and the theoretical sphere/spheroid the projections are based on. Get this wrong and your results might "look" OK but be meters off - you might not see it visually but it will cause problems for high-precision analysis.

I only have a little bit of GIS abilities but I found the page interesting. I think it's suitable for people as long as they have non-zero exposure to geographic coordinate systems.

One thing about this topic that makes it hard to learn is software ergonomics. It seems like a lot of tools can work with GeoJSON now, and as I understand it GeoJSON has simply declared that there is only one coordinate system and every file uses that one. Declaring a different CRS is not allowed in the standard. However, ESRI ArcGIS will emit GeoJSON with any CRS you want. In fact if you don't tell it specifically to use the GeoJSON standard, it will instead emit GeoJSON in EPSG 3857, which is not just irritating but plainly incorrect. It further confuses the issue that the 3857 system has six other alternate names, each seemingly as well known as any of the others.

It doesn't help that geopandas will cheerfully perform a spatial join of two dataframes despite the fact that it knows full well that the two frames have incompatible coordinate reference systems. It will warn about this condition but it will do the join anyway and you might not see the warning after a million lines of console spew.

Embrace, extend, extinct ... ESRI arcGIS may I be suspicious

Yeah the software ergonomics is the challenging part. It's all tradeoffs. In the case of geopandas, would you rather:

a) have the software (silently) transform your data, making decisions on your behalf about which are the source and destination projections, datum transforms, raster resampling, line densification, and precision.

PRO: easy, CON: potential poor performance, accuracy.

b) fail because the coordinate references systems don't match.

PRO: logical, type-safe, CON: Comparing coordinate systems is not a solved problem. You can have two identical (or practically identical) systems that are not equal by their representation or your programming language semantics.

c) try to naively attempt the spatial analysis in 2D cartesian space.

PRO: does what it says, leaves the responsibility to the programmer. CON: does what it says, leaves the responsibility to the programmer.

I've seen justification for all three approaches but no obvious solution. I tend to go with c) ala geopandas since ultimately the responsibility lies with the person building the system.

I think Erlangs approach is valuable: fail early and loud.

There is no harm done by b), there will never be a costly mistake. If the programmer has to deal with it anyways, at least make them very aware of the fact, they can then still decide to perform c).

Definitely this. When software silently thinks it knows what I wanted and silently does the wrong thing, sometimes you find out a long time later, and it ends up being very expensive, particularly when it’s to do with data analysis. I would always prefer an error.

I think this is a huge weakness of Pandas. I have to put “errors=raise” all over the place, although not many functions have that parameter.

That's an excellent summary.

I think just about every field in Computer Science has these kinds of problems though, where people learn just enough to be dangerous.

E.g.: the default sRGB colour space used by typical computers is non-linear, so you can't interpolate using naive algebra. You should convert to linear light, interpolate, then convert back to sRGB with the original gamma curve.

Practically nobody does this.

Even Adobe PhotoShop has this feature off by default which is just insane.

Just as with geospatial mappings, there are further nuances to imaging colour spaces. E.g.: Most colour spaces are designed to match some phosphor or LED colour, and do not match the human eye's response curves. So almost any "maths" you perform on almost any colour space, even in linear light will introduce colour shifts or other artefacts. E.g.: changing intensity, upscaling, downscaling, blending, etc...

A much better colour space would be to use something like ICtCp, which is designed to match the response of the human eye, so that the "coordinate system" would map better to the actual responses of the rods and cones: https://en.wikipedia.org/wiki/ICtCp

Note how the map of the colours is smooth, because it allocates "coordinate vector bits" to perceptual colours evenly.

This is one of the key features of Dolby Vision: https://professional.dolby.com/siteassets/pdfs/ictcp_dolbywh...

(Dolby is just about the only organisation that gets this stuff even vaguely right.)

All of the above issues can turn up in simple Long/Lat coordinates. There's always some idiot that ignores the fact that the coordinate density changes with location, or that the cells aren't even rectangles near the poles, or that interpolation is not trivial, etc...

I thought it was practically impossible to have a smooth color space because the real human color space is non-euclidean ?

So is the surface of the Earth. That's the point!

May I ask, how does one waste “hundreds of thousands of dollars” with a poor choice of coordinate system? That sounds spicy.

If the simple planar distance and the geodetic distance are significantly different and you use the planar distance for some analysis then the best case will be you got your analysis wrong, the worst case would be that you have bought the wrong quantity of material or built some unworkable or unnecessary. Or you just misplace where poles are and work crews waste time finding them, or you give wrong information for call before you dig queries and somebody goes straight through a pipe or conduit.

GIS systems are an area where software and the physical world can meet in ways many devs don’t appreciate.

I used to work for BT a while back and on the "Engineering" side there where many problems with the drawings of plant being wrong or lost.

Also workmanager aka workmangler was known for sending linemen (engineers in BT speak) to the wrong place and also sending multiple teams to the same fault

Yeah, I used to work on the Smallworld GIS software and it was fascinating seeing the hoops customers would jump through to deal with data having systemic problems. Some had to keep using there own background data because it was wrong, but it was also what everything had been entered into the system relative to. So if they started using correct background data they would have had to move all their assets to the proper positions, and the offsets were not universal across their operational area.

When you pull the trigger on a processing job involving a few hundred GBs of data and get a parameter wrong, you may have to re-run it and pay for the wasted compute/bandwidth. In this sense, getting the coordinate system right is no different than getting any other parameter right.

When you increase bad inputs by 1000x, the magic of scalable cloud infrastructure multiplies your mistakes for you automatically.

You discover the error too late and fail to submit your proposal. I've seen this happen for a 7 figure bid. This was in orgs that are not used to coordinate systems, for reference we have thousands of crs in my local area, sure when you know that you need so many you usually have the experience to get it right.

Don't forget NURBs

Reply on news.ycombinator.com