I Hate Coordinate Systems | Hacker News Reader

I Hate Coordinate Systems

ihatecoordinatesystems.com

365 pointsedent95 comments

The problem with a self-described "problem-based guide" to coordinate systems is that it relies on the user to identify the problem in the first place.

Reality is that these are expensive errors that can be avoided by systematic planning but made far too easy by default software options. I've seen weeks/hundreds of thousands of dollars wasted by proceeding with data analysis without properly considering the coordinate reference system.

If spatial accuracy is a concern and

- your data is vector-based lines or a polygons, there is no guarantee that the edges between your vertices will follow the same trajectory in a different coordinate system. This means that the spatial relationship (intersection, overlap, etc) between two geographic objects depends on the coordinate system and the relative density of points.

- your data is raster-based, you need to resample pixels values to translate between coordinate systems. If your pixels represent absolute/measured values (eg population), you need to consider this carefully.

- In both vector and raster, you need to deal with dateline and pole issues. The earth is not a cartesian plane.

- In both cases, you also need to consider more than the projection. You also deal with datums which define the relationship between the actual earth surface and the theoretical sphere/spheroid the projections are based on. Get this wrong and your results might "look" OK but be meters off - you might not see it visually but it will cause problems for high-precision analysis.

I only have a little bit of GIS abilities but I found the page interesting. I think it's suitable for people as long as they have non-zero exposure to geographic coordinate systems.

One thing about this topic that makes it hard to learn is software ergonomics. It seems like a lot of tools can work with GeoJSON now, and as I understand it GeoJSON has simply declared that there is only one coordinate system and every file uses that one. Declaring a different CRS is not allowed in the standard. However, ESRI ArcGIS will emit GeoJSON with any CRS you want. In fact if you don't tell it specifically to use the GeoJSON standard, it will instead emit GeoJSON in EPSG 3857, which is not just irritating but plainly incorrect. It further confuses the issue that the 3857 system has six other alternate names, each seemingly as well known as any of the others.

It doesn't help that geopandas will cheerfully perform a spatial join of two dataframes despite the fact that it knows full well that the two frames have incompatible coordinate reference systems. It will warn about this condition but it will do the join anyway and you might not see the warning after a million lines of console spew.

Embrace, extend, extinct ... ESRI arcGIS may I be suspicious

Yeah the software ergonomics is the challenging part. It's all tradeoffs. In the case of geopandas, would you rather:

a) have the software (silently) transform your data, making decisions on your behalf about which are the source and destination projections, datum transforms, raster resampling, line densification, and precision.

PRO: easy, CON: potential poor performance, accuracy.

b) fail because the coordinate references systems don't match.

PRO: logical, type-safe, CON: Comparing coordinate systems is not a solved problem. You can have two identical (or practically identical) systems that are not equal by their representation or your programming language semantics.

c) try to naively attempt the spatial analysis in 2D cartesian space.

PRO: does what it says, leaves the responsibility to the programmer. CON: does what it says, leaves the responsibility to the programmer.

I've seen justification for all three approaches but no obvious solution. I tend to go with c) ala geopandas since ultimately the responsibility lies with the person building the system.

I think Erlangs approach is valuable: fail early and loud.

There is no harm done by b), there will never be a costly mistake. If the programmer has to deal with it anyways, at least make them very aware of the fact, they can then still decide to perform c).

Definitely this. When software silently thinks it knows what I wanted and silently does the wrong thing, sometimes you find out a long time later, and it ends up being very expensive, particularly when it’s to do with data analysis. I would always prefer an error.

I think this is a huge weakness of Pandas. I have to put “errors=raise” all over the place, although not many functions have that parameter.

May I ask, how does one waste “hundreds of thousands of dollars” with a poor choice of coordinate system? That sounds spicy.

If the simple planar distance and the geodetic distance are significantly different and you use the planar distance for some analysis then the best case will be you got your analysis wrong, the worst case would be that you have bought the wrong quantity of material or built some unworkable or unnecessary. Or you just misplace where poles are and work crews waste time finding them, or you give wrong information for call before you dig queries and somebody goes straight through a pipe or conduit.

GIS systems are an area where software and the physical world can meet in ways many devs don’t appreciate.

I used to work for BT a while back and on the "Engineering" side there where many problems with the drawings of plant being wrong or lost.

Also workmanager aka workmangler was known for sending linemen (engineers in BT speak) to the wrong place and also sending multiple teams to the same fault

Yeah, I used to work on the Smallworld GIS software and it was fascinating seeing the hoops customers would jump through to deal with data having systemic problems. Some had to keep using there own background data because it was wrong, but it was also what everything had been entered into the system relative to. So if they started using correct background data they would have had to move all their assets to the proper positions, and the offsets were not universal across their operational area.

When you pull the trigger on a processing job involving a few hundred GBs of data and get a parameter wrong, you may have to re-run it and pay for the wasted compute/bandwidth. In this sense, getting the coordinate system right is no different than getting any other parameter right.

When you increase bad inputs by 1000x, the magic of scalable cloud infrastructure multiplies your mistakes for you automatically.

You discover the error too late and fail to submit your proposal. I've seen this happen for a 7 figure bid. This was in orgs that are not used to coordinate systems, for reference we have thousands of crs in my local area, sure when you know that you need so many you usually have the experience to get it right.

That's an excellent summary.

I think just about every field in Computer Science has these kinds of problems though, where people learn just enough to be dangerous.

E.g.: the default sRGB colour space used by typical computers is non-linear, so you can't interpolate using naive algebra. You should convert to linear light, interpolate, then convert back to sRGB with the original gamma curve.

Practically nobody does this.

Even Adobe PhotoShop has this feature off by default which is just insane.

Just as with geospatial mappings, there are further nuances to imaging colour spaces. E.g.: Most colour spaces are designed to match some phosphor or LED colour, and do not match the human eye's response curves. So almost any "maths" you perform on almost any colour space, even in linear light will introduce colour shifts or other artefacts. E.g.: changing intensity, upscaling, downscaling, blending, etc...

A much better colour space would be to use something like ICtCp, which is designed to match the response of the human eye, so that the "coordinate system" would map better to the actual responses of the rods and cones: https://en.wikipedia.org/wiki/ICtCp

Note how the map of the colours is smooth, because it allocates "coordinate vector bits" to perceptual colours evenly.

This is one of the key features of Dolby Vision: https://professional.dolby.com/siteassets/pdfs/ictcp_dolbywh...

(Dolby is just about the only organisation that gets this stuff even vaguely right.)

All of the above issues can turn up in simple Long/Lat coordinates. There's always some idiot that ignores the fact that the coordinate density changes with location, or that the cells aren't even rectangles near the poles, or that interpolation is not trivial, etc...

I thought it was practically impossible to have a smooth color space because the real human color space is non-euclidean ?

So is the surface of the Earth. That's the point!

Don't forget NURBs

You haven't seen pain until you've had to deal with crappy embedded systems that output "decimal DMS" data - i.e. ones in which 126.5904 means 126° 59' 04, rather than 126° 35' 25.44" as you might expect. Plus, the crappy documentation doesn't bother to tell you what coordinate system they're using so you get to guess! I wish I'd known about the little calculator they have on this site - which guesses the coordinate system in use - back when I was doing that project. Would have saved me a lot of headaches.

The next pain point has got to be China's insane "GCJ-02" obfuscated coordinate system, as alluded to and linked on this site. I've had a lot of trouble getting basically anything GPS-related to spit out meaningful things while in that country (more accurately, matching basically anything up with maps while there). While it has been reverse-engineered, you always get to wonder whether the authorities are going to come after you for having GPS data that's too accurate...

Coordinate systems are hard. GIS stuff is hard. I'm glad this website exists. Even if it's hard for people to identify the problem, this site at least makes one aware of the problems in the first place.

Ostensibly, GCJ-02 is for national security.

Given that all of China's potential enemies have detailed satellite maps of the region and aren't in the slightest bit inconvenienced in their targeting accuracy because of GCJ-02, that's obviously false.

Clearly, GCJ-02 is just simple, ordinary corruption, the type endemic to China. Someone convinced some politician with a bag of money to keep a legally mandated monopoly on mapping, locking out foreign competitors.

PS: Japan does similar things, where for example Google Maps refuses to "download" maps of Japan because by law they are not allowed to make their own maps of Japan and must license the data from a local provider. That local provider does not allow offline use.

The world of mapping is nuts.

Is it illegal for Google to make maps of Japan or just not a good investment?

(that the licensed data they use has some restrictions is clear enough)

Seems to have been more of a licensing contract things, apparently it got fixed right after my most recent trip to Japan:

https://www.reddit.com/r/japan/comments/dt4pe7/google_maps_a...

South Korea is weird too, Google Maps only have pre-rendered bitmaps from local provider.

Yes, South Korea has this braindead law prohibiting "export" of map data - so in order to have a map service the map data server must be physically located inside Korea.

> Given that all of China's potential enemies have detailed satellite maps of the region and aren't in the slightest bit inconvenienced in their targeting accuracy because of GCJ-02, that's obviously false.

What makes the situation even funnier is that Google-Earth-style satellite images were (and still are?) not regulated as maps, and they're free to be published with unobstructed WGS-84 coordinates. Only the maps are regulated to use GCJ-02. If you use a standard GPS receiver, your location will be perfectly normal in satellite view, until you switch the display to map view. Claiming it can protect "national security" is just fooling oneself.

The tragedy is that good implementations are rare because correct implementations are really difficult. Everyone grossly underestimates how difficult it is to implement precise, correct behavior globally in software without weird emergent problems.

Odd edge cases are the default state of reality for geospatial unless you are extremely careful. Even if you are diligent and assume a global high-precision ellipsoid as your Earth model, you still have to ensure that the computational geometry is correct in that context for every possible use case. Operations on real surfaces are difficult to implement on discrete computers.

As a simple example, correct ellipsoid computation in double precision sometimes can't even be computed in quad precision, never mind double, yet most efforts at computational geometry limit themselves to double precision because that is what computers support natively. Manipulating relationships in real-world coordinate systems is not for amateurs.

My impression has been that basically ESRI seems to have more correctness and the open source stuff has done the first 90% and left the second 90% on the table. What do you think of libraries like shapely, if you've tried it? I make maps of the county where I live and there are parcels that ESRI doesn't seem to mind but shapely refuses to ponder. An example:

1: https://drive.google.com/file/d/1Fw5SdZ-aAiHrZERnIkVgll_OFnt...

The open source stuff is mostly really fidgety and works correctly with a rather narrow set of inputs. To make most libraries be actually easily usable for a wide range of inputs is a tremendous amount of work.

The core of them mostly do the thing they claim to do, but if you feed them data that's just slightly off, then it'll break in interesting ways.

Speaking based on a lot of experience[1]. Nothing in the geospatial or environmental data world is easy and there will always be an unexpected problem lurking right behind the corner, regardless of how many of them you've already fixed or worked around.

Still, the complexity of the problem space also makes it fascinating. As an example, the definition and measurement of something that's seemingly very simple: mean seal level is actually tremendously complicated if you think about it. The Wikipedia article talks about averaging over decades of measurements to get it right for example.

[1] I'm a part of the team behind https://data.planetos.com/datasets

ESRI is more complete than the open source implementations generally, but is not usable for large-scale geospatial analytics. I have not used Shapely but it appears to be based on the same geometry routines as PostGIS. If you look at the underlying implementations of most open source for geospatial algorithms, of which there are only a few, it is usually missing code paths for complex numerical analysis edge cases regardless of what the documentation says. (Sometimes there are even code comments to that effect.) At sufficient scale, you will find all the edge cases in real-world applications quickly even though you may not notice if you have nothing to compare it to. When we developed the first “analytics grade” ellipsoid geospatial geometry engine several years ago (an enormous technical undertaking involving a team of mathematicians), it created distress at some companies because the visible discrepancies relative to other geospatial analysis systems they were using were much larger than they imagined.

A challenge of open source software is that it largely only gets created when the technical complexity and expertise barrier is low enough. Above that threshold, the small number of people capable of contributing combined with the large number of man-hours required for a correct implementation is beyond the resources typically available to such a project. Geospatial is a good example of this pattern (database engines are another).

What do you mean by double and quad precision?

I would guess 64bit floating point (double) Vs 128bit floating point. Eg f32 and f128

Probably 64-bit and 128-bit floating point numbers.

Refers to the number of "words", in this case 2 or 4. The term word is not used so much anymore because it's kind of lost its usefulness in modern computer architectures. It used to refer to the size of CPU registers and/or bus widths. I guess the last generation where this was still as important was 32 bit, and that kind of stuck, hence double word floating point.

[Edit] More informed description: https://en.wikipedia.org/wiki/Word_(computer_architecture)

Yes, these days when the typical word is 64-bit sized, these names have to go.

Okay, now do space reference frames.

https://naif.jpl.nasa.gov/pub/naif/toolkit_docs/Tutorials/pd...

https://en.wikipedia.org/wiki/Earth-centered_inertial vs https://en.wikipedia.org/wiki/ECEF

And then think about tectonic plate motions and that any point affixed to a continent is moving with regards to an ECEF coordinate system.

Ugh, I get flashbacks to one of my astronomy intro courses and having to memorize conversion formulas between horizontal (azimuth/altitude), equatorial (RA or hour angle/declination), and ecliptic (ecliptic long/lat) coordinate systems. Fun times.

While in uni I once ended up having to chase an autonomous boat as it decided to head out too deep waters and do a hoky-poky dance because one guy wrote controls code in NED coordinate convention and another one issued commands in ENU convention.

Write a blog post if you can. Chasing a boat on the water because the program used the wrong coordinate system - sounds like an amusing debugging story.

No kidding, this article is someone complaining about easy mode...

My main hatred is the simple one

Lat/Lon or Lon/Lat, and what about altitude? especially when libraries have parameters like x, y, z and you have to triple-check which value is going into which vaguely named, not well documented, field.

Because there are different conventions on which value goes to which depending on your background.

Another fun one is altitude vs elevation (then of course it comes down to whether you're using a WGS84 ellipsoid or something more esoteric)

Not to mention that different coordinate systems also use different orderings.

Good read on this: https://macwright.com/lonlat/

Can I add another, personal gripe to this? US surveys use their own definition of a foot, which is different from everyone else's : https://en.wikipedia.org/wiki/Foot_(unit)#US_survey_foot

The most common error I've seen in this area is that most GIS systems use (Lat, Lng) but most systems written by non-GIS people use (Lng, Lat), which they think of as (x, y).

I think you have that backwards? In my very limited experience most GIS systems use ENU ordering (east, north up), and easting corresponds to longitude.

I think he has that correct. Ime it's split on east/north vs. north/east but wgs84 is very dominant with lat/lon.

Huh, thanks. Researching this more, I think you're right.

I thought the Open Geospatial Consortium's standards would reflect dominant practice, that turns out to have been the wrong assumption [1]

The only geospatial software I've used is Proj4J, which normalizes with a function called toENU.

I guess I've only encountered weird stuff then.

[1]: https://lists.opengeospatial.org/pipermail/coordtran.wg/2006...

I'm working on a system that transform data from Long/Lat/Alt (WGS84) to a local tangent plane (LTP) East/North/Up coordinate system. To go from WGS84 to LTP, the coordinates first need to be transformed to Earth-Centered Earth-Fixed (ECEF) coordinates and then translated and rotated.

I was quite surprised to find that while it is straightforward to go from WGS84 to ECEF with a closed form, the other way, from ECEF to WGS84, needs to be computed iteratively [1] and is not trivial to find.

[1] https://en.wikipedia.org/wiki/Geographic_coordinate_conversi...

Quick tip from someone else who was working on similar stuff last week: there’s a wonderful paper by Bowring from 1976 that has a one-step “iterative” version of the conversion. To verify that it worked right, I compared the output with the LLA output from my receiver and it matched to as many sigfigs as the receiver output.

[The Author] Hates Geographic Coordinate Systems

Yeah, this wasn’t really what I expected but it was interesting nonetheless.

As someone who works in genomics, all these issues apply to genomic coordinate systems as well—same concepts, different words.

And as someone who works in CAD/CAM/robotics software, these issues apply to robotic coordinate systems, too: World coordinates, joint coordinates, polar coordinates (not the same as high-latitude GIS coordinates, think angle + distance instead of X and Y), tool coordinates, tool offsets, work offsets - all of those are just vectors of 3-6 numbers. Instead of esoteric international standards, and huge modern software packages, we have a lot of old machines, a lot of old code, and a lot more tendency towards 80s-style strategies: Post a sign stating "Do not dumb here!" and leave it all up to the operator to get it right.

There's nothing like telling your 6-figure CNC or multi-axis robot to make a piece of carbide occupy some space where you currently have precision-ground hardened steel because someone left a G92 somewhere you didn't expect it to be. Though I suppose that errors can be at least that expensive in GIS if you're quoting civil engineering or real estate...

As someone in computational physics, let me assure you it's the same there. The one thing is doing math right and having the correct idea about coordinates. The other is being able to tell the computer about these ideas! Many people cannot code, so are not aware of array or vector facilities of their programming language/ecosystem they use (or rather have to use, because it's always have been like that).

This Phd comic is pretty acurate about the situation: http://phdcomics.com/comics.php?f=1689

Now imagine being a newcomer with neither programming experience nor domain specific knowledge and asked to add some feature to such a legacy codebase. This is the standard case in science.

I feel seen :-)

In my first job I updated someones (doing a PHD in fluid mixing) Fortran code.

The original software to run an experimental rig just put up a ? prompt and you typed numbers in to drive the experiment.

You had to memorise which number was used for what and at which point you had reached.

I updated it to prompt the user and at every step display what options where available and also allowed you to go back a step.

I even went as far as to use Mixed Case Hollerith segments!

My thesis supervisor was surprised when I handed over my code in git repo, CDROM and USB drive.

Apparently they're used to just receiving print-outs shudders

Author here. I also hate projected coordinate systems!

But in all seriousness the domain name is channeling an attitude felt by generations of GIS practitioners. I don’t really know why it’s on HN though...

I love it. Thank you!

It's not your fault. HN users love ripping articles out of context, and it's against the rules to put some context back.

I think what this mostly shows is the unimaginable complexity that many parts of life carry, even if I myself may not have insight into them.

And it's probably an alright source for people whose geo data isn't looking right.

I try to keep things like this in mind when I have that "I don't get why don't they just..." thought about a topic I know very little about.

I love geographic coordinate systems. My one wish is if the GeoJSON spec as loosened so that it technically allows non lat/long coordinates. I use it all the time for indoor flatland scenarios where units are metres. It all works fine. But it's technically not allowed.

On rare occasion a GIS tool will completely balk at the data being CRS.Simple

Null Island is hardly the only problem with bad quality data. Quite often, when no exact location is known, but the country or region is known, the data may give coordinates at the center of that country or region, which can easily be hundreds or thousands of kilometers away from what you're looking for.

Case in point: https://web.archive.org/web/20171201234812/https://consumeri...

Man coordinate systems are hard, especially mapping between systems.

I had to do this for the backend of a game I'm building - built my own geohash so I could add some optimizations I wanted (like storing nearby hashes for certain nodes etc).

Created some fun graphics while testing the bounding box queries, like:

https://jvm-gaming.org/uploads/default/optimized/2X/d/d7c489...

This has really struck home.I'm missing another one.

Your dataset is tilted 90° and in some other part of the globe: you probably swapped latitude and longitude.

Tip: latitude sounds like "altitude" so it has to be the y coordinate

It's such misnomers. You'd think the "lat" prefix comes from lateral and means an axis that goes sideways, but that's not the case.

Not to mention longitude and latitude are most often used for spherical coordinate systems, as angles, where x, y, z are usually meant for Cartesian coordinates, as distances, in a projected representation.

As a French nitpick, the "sideways axis" is actually not going sideways, but pointing right at us :)

There's an unavoidable duality (An axis is definitionally perpendicular to the lines of constant value)

The core of the problem is that the system talks about "lines" too much. In regular math, we have axes, but we don't think much about a point being at the "intersection of the x=3 line and the y=5 line", it's just "horizontally 3 and vertically 5". But in GIS we visualize a full grid, with "lines of latitude" instead of "lateral deviation".

Very easy to do, especially if you are dealing with both lat/lon and other coordinate systems that may use a different order.

Clickbait, he doesn't hate coordinate systems, he loves them.

This sort of thing bites me every so often. A client wants something in KML, which I have used ... once? The export files look suspiciously small. Converting back gives zero results. No error on the export.

Mentioned nowhere is that you need WGS 1984 in the mix or you get nothing. That was fun.

The thing that annoys me as a software engineer with background in mathematics is that it's hard to find pure transformation formulae between these geodesic coordinate systems, or at least was 5+ years ago when I last played with it.

I get all the gotchas, but formula is much easier for me to get my head around rather than these articles which are centered around me having to learn a lot of terminology just to be able to do a transformation between coordinate systems. Even if that formula was accompanied with a correction table for particular use cases.

I expected this to be about coordinate systems used for video game models (y up vs z up). Turns out it's geospatial positioning.

Same here. I absolutely hate 3D coordinate systems, more specifically handedness and how easy it is to have correct results but wrong math due to two wrongs canceling each other.

Me too, confusing to transform from one to another, error prone and computationally awkward

I expected this to be another post about geometric algebra (or maybe tensor algebra)

Yes, the Earth is curved. Assuming it is flat is incompatible with having a robust and accurate geospatial search. You must use sines and cosines if you want to do geospatial indexing correctly!

Sines, cosines, and the Haversine formula only get you so far (or approximately close). The Earth is neither flat, nor perfectly spherical, but an oblate spheroid. The coordinate systems referenced in TFA are defined to adapt for this, but in different ways depending on the geospatial application, source data collection, map projection, and other factors. A reusable coordinate dataset will include not just latitude and longitude, but also the reference ellipsoid to which it approximates the Earth's surface, and the direction of North used at the location to calibrate sufficient dimensions to pinpoint relative location. Not in a word, ideal, and thus, TFA.

Mt Everest and The Marianas Trench would like a word with you about The Earth being an oblate spheroid...

Everything is an approximation.

True the Earth is not flat in many different ways, and also, the parts are moving.

Edit: if you can put either of those extraordinary places in touch with me, that's fine also.

Related xkcd: https://xkcd.com/977/

Back in the day.... I tried to stay in HARN as much as I could, or at the very least something local. There were some minor precision problems but never anything drastic.

Does anyone here know a good source that gives details on how to redefine coordinate systems without relying on existing software implementations?

Snyder is the canonical source that most use. Free download from USGS. Once you see the math you’ll understand why you want to use libraries.

https://pubs.er.usgs.gov/publication/pp1395

Same when I found out the need to use fortran for astro calc ...

HA!

The upshot of Snyder is that it includes -samples- for everything, which is really useful if you ever wind up having to deal with an Albers Conic over 3 states in the US, that also is using an elipsoid typically used in India.

Oh, also it's 2010 and you can't afford actual GIS tools. You have Google Earth, Visual Studio, AutoCAD, and Microstation. Good Luck.

The rather shouty title makes me wonder if someone put Descartes before the hoarse

Wow, thanks for posting this about geospatial Coordinate reference systems (CRSs)!

My two cents. Missed opportunity for the domain ihatecoordinate.systems

Here was me thinking this would be a primer on differential forms.

This is a really unfortunate title that seems like it might be interesting to general math-inclined people, but instead is only interesting to a very small group of engineers working in a very specific domain.

You’re totally right, I made it for GIS people, not general math people. No idea why it’s on HN tbh

Lots of GIS people on HN it seems. My favorite projects were always raster based model projections for various infrastructure and demographics.

As someone who isn't a GIS person but is trying to make a map-centric app (hiking/backpacking, with activity tracking and route planning), any suggestions for resources?

I just finished implementing enough of the WMTS spec interact with the one provider I'm using so far (UK ordinance survey). The only library I could find to help that works on Android was proj4j, which I'm using to handle converting points to different CRSs.

I'm making incredibly rookie mistakes like just realizing I need to project vector data into the same crs as the raster base map.

I want to be able to transparently support changing the base map to one with a different crs and lines and points recorded in the previous crs show up in the "right places", but I can't figure out how that's possible. I think I've seen other apps do it.

I don't work anywhere near geospatial stuff, but this would have been helpful to me a few years ago when I did some hackathon projects that involved Google Maps, frantically trying to integrate a number of different data sources that used coordinate systems I'd never heard of. It's worth having at least a vague awareness of this stuff if you might ever touch anything map-related.

Reply on news.ycombinator.com