Comment by wpietri | Hacker News Reader

Comment by wpietri | Hacker News Reader

In contrast to his 4-digits-are-fine notions, let me offer a counterexample: http://scissor.com/transient/destination_heatmap/

I am working on a Twitter bot called @sfships, which monitors the comings and goings of large ships in the San Francisco Bay: https://twitter.com/sfships

As part of that, I generated the above map from AIS data. [1] It's basically where ships stop. If you zoom in on the San Francisco waterfront, you will see a grid of dots. That's because the AIS protocol stores lat/lon as minutes/10000 [2], throwing away more detailed information.

This is adequate for its initial intended purpose, which is putting ships on radaresque displays so that ships don't hit one another, etc. But it produces all sorts of artifacts and issues when one tries to use the data more broadly.

And in case anybody is interested in playing with this, I have written a python parser for AIS data with a bunch of Unixy command line tools, including one that just turns the weird 90s protocol into more modern JSON: https://github.com/wpietri/simpleais

[1] https://en.wikipedia.org/wiki/Automatic_identification_syste...

[2] https://gpsd.gitlab.io/gpsd/AIVDM.html#_types_1_2_and_3_posi...

I used to have a job where I analyzed global ship tracking data (AIS and other less-open (i.e. more classified) data streams) for law enforcement and environmental conservation purposes, and learned one thing: man is AIS data a bitch.

This is especially true when you're dealing with potential bad actors who can spoof their transmissions, combined with incredibly poor vessel metadata databases and strange satellite AIS coverage issues.

If there are any data scientists out there looking for a real head-basher, start an AIS project.

Why do bad actors spoof transmissions? Smuggling?

I'm sure Tony knows more, but one I've heard about is fishing. It's not legal to just fish anything anywhere. Poachers have an incentive to avoid monitoring and enforcement.

Many reasons, and a significant percentage of all AIS transmissions are dodgy. Smuggling is just one reason.

Other common reasons include illegal fishing in protected waters, covert meetings (e.g. a couple yachts coming together in the middle of the ocean), spy ships by state actors trying to look like ordinary commercial traffic, and misrepresenting state of shipping assets for commercial leverage in contract negotiations.

I would love to read more about this. Any suggestions?

These are things you learn by doing in-depth analysis of AIS and related data sources. As far as I know, no one writes about it. Physical world data sources are full of interesting patterns and anomalies. Few people ever look because the data sources are challenging to work with and very different from Internet data, which is the only kind of data most people are used to.

Well at this point I have terabytes of AIS data from the last few years, so if anybody is looking to dig into this, I'm glad to share.

Much of this kind of thing is written up in pretty terribly-written documents that come from government/law enforcement. Here's [0] a report I helped to put together which circles around some of these issues.

[0]: https://ocean.csis.org/spotlights/illuminating-the-south-chi...

Yes, these are all things we observed in our data.

That sounds really interesting. And if anybody is doing an AIS project, feel free to get in touch, whether or not you're using my library. I have soaked up all this esoteric knowledge that I'm eager to pass on.

Another example: I was looking up my property details yesterday, and wanted to get exact coords for my property line. Their web app lets you click on the map to get coords, but it only displays degrees/minutes/seconds (with no decimal point), so it wasn't much use.

Are you certain that the data is accurate enough to support better precision? Where I live we have avcess to vectorized property maps, but the data source may be a digitized 200 y/o map.

My mom used to do a lot of work with property maps for oil and gas companies and at least in the US they were very precise. They had to be, as you don't want to build an expensive well and then later have to move it 3 feet. Or, worse, realize you didn't buy all the mineral rights and then have to pay somebody.

> If you zoom in on the San Francisco waterfront, you will see a grid of dots.

And yet, some of those dots are significantly further inland than rounding can account for. There's one faint point near 4th and King, for example, and a strong one on Spear between Howard and Mission. All signs point to the source data being imprecise enough that transmitting it at a higher precision wouldn't improve the results.

I wonder if introducing some randomized jitter would combat the regular/grid-like placement of the dots.

Reply on news.ycombinator.com