High Precision Mapping from Scratch | Hacker News Reader

High Precision Mapping from Scratch

bendauphinee.com

141 pointsbendauphinee46 comments

An interesting case study in measuring one's own land.

I have a former working background in surveying calcs and mapping, and have also been long considering embarking upon a similar DIY project for my own property.

One aspect I note, having spent quite some time at it myself, is that there is somewhat of an art to the generation of accurate topographical contour lines from a set of manually surveyed points. Both in what points to survey, and in how to treat the points. The reference term here is TIN or Triangulated Irregular Network. Automated software does a fair job of building TINs, but the resulting contour lines can in some areas represent detail or features that don't really exist.

For example in the author's contour maps under the "Finally, The Mapping" section, much of the leftmost half is particularly sparse in measured points and as a result, the contour lines have interpolated some wacky humps-and-hollows across that half of the land, at least some of which probably don't exist in reality. (Technically speaking, long skinny triangles = not ideal, particularly at large scale; and separately, another key word or concept to look into is that of topographical "breaklines")

Ultimately, the specific arrangement of triangles in the TIN defines how the contour lines will be drawn. Good starting data is half the equation, and the other half is how you TIN it.

Yup! And the TIN Mesh Creation in QGIS was the first way I played with to model the terrain too, though I was less happy with how it looked. It's useful for me to see where the data is too sparse, but not a great view of the land itself.

You're correct, my data is still too sparse in some areas to give a good set of contours. It's been fun to see how the contours change as I add more data to the survey. It's part of the reason I decided to buy my own gear, because then I can spend as much time collecting data as I want without more cost.

This old forum thread details a similar project I did (on the data side) for somebody about 10 years ago.

https://forums.overclockers.com.au/threads/calling-all-sketc...

Quite outdated now in many respects - and using work software worth thousands, at the time! (modern QGIS is great) - but you might find some of the crossover detail interesting.

I've added a TIN mesh illustration to the bottom of my post, to illustrate this point as well.

Database connection error - could not find the article on archive.is or archive.org but it was available on Google Cache:

https://webcache.googleusercontent.com/search?q=cache%3Ahttp...

Great writeup, may add to my list of projects for 2024.

Yea, I just got notifications my little VPS fell over, I've rebooted it :D

My apologies to anyone having trouble accessing my site. My little VPS is getting smoked, because it's a completely untuned install of Wordpress on a very small server :)

I am halfway done reading, "GPS and GNSS for Land Surveyors Fidth Edition", which I bought to better understand the survey issues we were having with surveys and data on a project I was on in Saudi Arabia for 6 months last year and 2 months this year. I was a physics major and I have to say surveying is a lot more complicated than I had ever imagined. Realizing that relative height data from one point to the next is straightforward, but that a simple elevation reading is based on MSL and what area in the world you're in for your reference are just a few of the variables that go into the craft of surveying. I really went down the rabbit hole, but I've learned so much and I am now hooked on taking it much further.

In my high school in 1980, we took a 2-hour survey/questionnaire filling in circles with a #2 pencil, and we we waited 3 to 4 weeks to get the results of what career we were suited for, and mine was cartographer! I imagine there were humans and machines doing the analysis back then, but maybe I should have followed that path!

Another good source for RTK receiver stuff could be https://www.ardusimple.com/

I was once going through the same process and considered buying from them. But then a more official survey turned out to be rather cheap so I did not have any need any more.

https://bendauphinee.com/writing/ Link to the post

It's an interesting technical challenge. I wonder if there is any "tangible" benefit by moving your land these few meters and spending $4600 on a surveyor to do it considering it's a woodland property. One immediate benefit I can see is that it now borders a road and previously it didn't. That's prety useful.

What are the laws in that place? Could you, let's say, not do any surveying, build a fence, and then have a new neighbour show up 25 years later with a survey and ask you to move it?

Here, we have this law that if you occupy some piece of land for 20 years (with no "malice", if it can be proved you knew it wasn't your land the amount of time is 30 years). After that time it essentially "becomes yours". The neighbour had no fence, but I imagine he could argue his camper being there for the required amount of time fulfills the requirement and then I imagine the boundary would get redrawn.

What is the authoritative thing that defines the boundary in absence of pins/markers?

For me, the tangible benefit is in actually knowing where my land is. I have some things I want to do on it, and I'd rather find out before starting, rather than in the middle of building.

The authoritative thing is the boundary descriptions for your purchase (See my comment https://news.ycombinator.com/item?id=38829732 for what that looks like)

How did the neighbour take the new line in the end? Did he 'lose' land to you or did the boundary between your parcels move outward of his (presumed) land?

I had a conversation based on the GeoNB borders, and we had basically come to the conclusion that neither of us were going to be nasty about things. He was still annoyed to find out the realtor told him wrong, and he wasn’t in the middle of his property like he wanted.

With the official survey, his camper pad became officially on his land, instead of maybe on mine, so we’re both happy about not dealing with that.

2 things:

first, absolutely fascinating topic, and solid write up. i skimmed through certain bits but am wondering: "Turns out, the survey was off a lot more than any of us expected" did you mean off from the title, or the surveyor's work? (also, since you're in Canada, allow me to quote Les Stroud: "if you sweat, then you die" sweating is actually the easiest way to get cold heh, you brought sweat up in the same paragraph)

second, i'm a musician who lives in Anchorage. i'm a recovering programmer. a good friend of mine is a recovering surveyor, and he runs Trapper Creek music festival up here, 15 years and counting. since you said you mounted equipment on a mic stand you just happened to have -- wondering if that event you were planning for was music related :-D

> did you mean off from the title, or the surveyor's work

The data from GeoNB was way off from what the surveyor properly located.

> wondering if that event you were planning for was music related

There may be a band, but I just happened to have an old mic stand handy for that temporary test :D

This was wonderful!

Do you have more posts about how you went about buying the land?

The surveying aspect is super interesting, including the fact that your neighbor had such an imprecise knowledge of their boundaries.

I wonder what the history of surveying looks like... I assume as mapping precision increased there must have been many many cases where structures were found to lie on neighboring properties.

I'm not a historian of real property law, but my understanding is it's more often the other way around. The legal boundaries were/are defined by a mixture of reference to objects on the ground and conventional use. Any WGS84 coordinates you might have were non-normative. The concept of land ownership predates the concept of a single unified coordinate system by millennia.

I assume you kind of start with boundaries defined by physical features "between this river and than hill" or similar. Then you get a secondary level defined based on this. So "X meters from the top of this hill".

And I assume now property boundaries are kind of based on a mixture of all of this? But there's a trend toward WGS84 coordinates or similar?

I also assume along the way measurement mistakes have been reasonably common or situations where those "stable" features in the landscape turned out to be not so stable.

Resolving all these issues must be a nightmare and while I've been vaguely aware of this history of surveying it now seems like it must be quite fascinating and I should try and read more about it.

If it's the history of surveying land for land use then it starts in documented history with the Egyptians marking out growing land that floods every year and was restaked with string and sight from above the flood level monuments.

If it's cartography, the two were intertwined (with differences).

Before WGS84 there were tens of state level ellipsoids and hundreds of global datums - with more than a few readjustments on the way to global unification.

One book, of many on the cartography side is

Flattening the Earth Two Thousand Years of Map Projections - John P. Snyder

https://press.uchicago.edu/ucp/books/book/chicago/F/bo363285...

who formulated the sinusoidal US space projection seen in war room move scenes showing global maps with S tracks across them for satellite paths.

On the dull and technical side (a lot of truncated Taylor series and a couple of projections using <gasp!> complex numbers) see:

Map projections: A working manual - also Snyder (free download for PDF & wallchart)

https://pubs.usgs.gov/publication/pp1395

Fun fact, we actually use the NAD83 coordinate reference system (CRS) in North America. This is a plate fixed CRS, that kind of mostly can map to WGS84, but will be different in places due to continental drift and other factors.

Here's my actual parcel description:

All that certain lot, piece or parcel of land situate, lying and being on the southeastern side of the XXX Road, in AREA NAME, in the Parish of PARISH, in the County of COUNTY and Province of New Brunswick bounded and more particularly described as follows:

Beginning at a point on the southeastern boundary of a Crown Reserved Road known as the XXX Road; said point of beginning being located at a distance of ONE DISTANCE NUMBER (###) metres measured southwesterly along the southeastern boundary of the XXX Road from its intersection with the southwestern boundary of the OTHER PROPERTY OWNER NAME property by document number ##### registered in the COUNTY County Registry Office in book ### at page ###; said boundary being the southwestern grant line of Lot # originally granted by the Crown to PERSON NAME.

Thence in a southwesterly direction following the southeastern boundary of the XXX Road a distance of ONE DISTANCE NUMBER (###) metres to a point at a corner at the intersection of the northeastern boundary of the PROPERTY OWNER NAME property by transfer number ###### registered in Land Titles for the District of New Brunswick on DATE.

Thence in a southeasterly direction parallel to the southwestern grant line of Lot #, the OTHER PROPERTY OWNER NAME property following the northeastern boundary of the PROPERTY OWNER NAME property a distance of THREE DISTANCE NUMBER (###) metres to a point at a corner.

Thence in a northeasterly direction on a line parallel to the southeastern boundary of the XXX Road a distance of ONE DISTANCE NUMBER (###) metres to a point at a corner.

Thence in a northwesterly direction parallel to the southwestern grant line of Lot X, the OTHER PROPERTY OWNER NAME property a distance of THREE DISTANCE NUMBER (###) metres to the point of beginning.

The so-described parcel of land contains an area of SOME (###) hectares more or less and is located in Lot # originally granted by the Crown to AN ORIGINAL OWNER.

BEING the same lands and premises (PARCEL NAME) conveyed to LTD CORP by Transfer duly registered at the Provincial Land Registration Office in the District of New Brunswick on DATE, as number #######.

Fascinating, thank you!

I find it interesting that the boundaries are defined by a combination of physical land features and other property boundaries.

I guess this means that there's the error could potentially accumulate when you reference against property boundaries. Then you come up against some physical reference and have to resolve the inconsistency somehow.

And on top of the fact that a lot of boundaries outside the city have been low resolution for a long time (land grants may have been from like.. 1902 for some farmer with 400 acres, and then just parceled over time with no further survey). The fact that I shifted everyone 140' by pinning my one line shows how it can be, and I'd bet there are areas that are worse because they're less developed / further out / nobody cared to ask).

TBH, looking into the history of surveying in the US is fascinating. The level of accuracy they were able to achieve over such long distances so long ago is awesome. I fell deep down this rabbit hole when I started flying a LIADR unit on a drone and working with PLS.

Much of your local mapping is based on monuments and triangulation to those monuments. The fact that the monuments are now frequently stable GNSS base stations helps tie everything together.

This is an awesome project. Curious how much we could "cloud" apply NTRIP RTK corrections over 5G to any mobile phone device. It wouldn't be survey quality but it seems like it would work well enough for many use cases.

You'd have to have a GPS capable of taking the correction data, which a lot of units can't. I'd be interested if the iPhone could, since it has L1+L5 capacity, but I'd bet not, and iOS doesn't expose their location data very well anyway (a lot of mapping seems more flexible on Android for this reason).

Right now, the GPS receivers that support RTK corrections charge a substantial premium for the capability, so you're unlikely to see this in a mobile phone without the phone makers intending it.

Also, you only get ~3cm precision while you have direct line of sight to a reasonable number of satellites, including some that are closer to the horizon. So if you're close to a building, or in a car with a metal roof - the accuracy might not turn out that great anyway.

People doing fancy surveys use a special antenna that rejects reflections from near ground level, but has to be pointed towards the sky. That's undesirable for a phone, where the user might want it in portrait or landscape, might hold it at an angle for readability, and so on. I have no idea if you could achieve respectable RTK performance with an omnidirectional antenna.

So even if you were the iPhone GPS chip selector, and you had the market leverage to get RTK support cheaply and get a killer deal on network NTRIP corrections, you might think the benefits aren't worth it.

I suspect costs will fall precipitously in the next few decades, though, because this sort of tech is useful for self-driving cars, and automotive order volumes tend to make chip prices fall a great deal.

Google has held a couple of GPS precision challenges for smartphones, and Galileo recently launched their High Accuracy Service (HAS) for high accuracy Precise Point Positioning. Android has built in support for full GNSS phase measurements. Smartphone positional accuracy is already pretty good with L1+L5 bands, and it will probably get even better soon.

https://www.kaggle.com/competitions/smartphone-decimeter-202...

https://www.euspa.europa.eu/european-space/galileo/services/...

https://barbeau.medium.com/gnss-interrupted-the-hidden-andro...

Should perhaps start with universal SBAS support first, which seems to be not a given still.

OTOH at least in Europe there’s the EUPOS project that, if opened to the public, could very well provide good-enough corrections to give everybody high quality results.

I wonder what the gain is in doing this compared to using government lidar data - it seems this is available for new brunswick [0]

The 0 cost solution would be to use this lidar data for topography, perhaps even for tree clusters, then use an aerial photo to help plot on some extras like roads. Maybe there's a little accuracy difference I'm not sure

[0] https://open.canada.ca/data/en/dataset/80ccc975-d6ec-9e24-a7...

There's a lot of ground clutter for starters. A lot of what I've surveyed so far has been ground under dense deadfall (due to previous logging), and my resolution is better than the LIDAR, for those spots.

Also, the accuracy, I expect that since I can get precise measurements, my maps of the streams, marshy areas, and other features are going to be a lot more comprehensive. Basically, I can get detail down to the level I want, just for the time I want to spend putting stick to ground.

Good points! It would be very interesting to see the comparison

I'd be quite eager to see the elevation comparison (residual) between LiDAR and your surveying.

In my game (geophysics) LiDAR is basically known for its ability to see through canopies -- through the virtual holes between leaves. They define the last arrival as "ground" level, and call it a day.

Now, I agree that deadfall (i.e. tree bits-and-pieces on the ground) has an ambiguity such that the tops of the deadfall would be reported as "ground" in the LiDAR definition. So, I expect that any residual map you might be able to create would -- in essence -- show the deadfall "topography".

Very nice project!!!

I had not tried pulling the LIDAR, but yes, especially where the deadfall is 3-8 feet thick in the logged areas, I can already see spots where their generated topo doesn’t match my work.

If you want to go further down the LIDAR path, you might be able to improve the public data by making manual improvements to the point cloud classification.

For that sort of terrain it's common for classification algorithms to put vegetation (e.g. deadfall, small bushes) into the "ground" class, so when the terrain model gets triangulated using all the "ground" points there is erroneous bumps. If this is the case, you could reprocess the point cloud yourself and tweak the settings being used or make manual adjustments.

I'm not sure of the quality of the linked dataset, so maybe it would be difficult to find improvements. But you knowledge of on-the-ground conditions (and smallish scale of the area) means it's definitely possible.

Also, just in case you haven't stumbled across it, a "Digital Surface Model" (DSM) is different than a "Digital Terrain Model" (DTM)!

(surveying student here, very much enjoyed the writeup!)

I had not come across the DSM term yet, thank you! I'll have to pull that thread once the snow makes this work completely impossible for the season :D

A really nice write up! I’m a professional land surveyor and it was interesting reading an outsiders perspective; for the most part you really nailed it!

Thanks for the feedback! I'd be glad to hear anything I missed on, or ideas for improvement, since this kind of thing is very much a seat of the pants learning experience :D

I'm curious if you looked into PPK and possibly hiring someone with a drone based LIDAR setup to give you a quick DEM.

I actually prefer PPK over RTK for all work these days. RTK still gets used, but only to help control the drone's flight path in specific scenarios. Otherwise, uncorrected flight paths are fine since all the collected data is run through PPK after collection.

I had thought about PPK, but I like the reassurance of knowing I got a clean point on site, especially given tree cover. I also considered drone work, and I'll likely put hands on a drone or someone who has one in future, but not for LIDAR, just mapping in general.

Since my goal isn't just getting terrain, but also mapping features in detail (I'm doing site planning for an event as well), I may as well skip the drone LIDAR because I'm going to be walking the ground anyway and I can do better than it can, was my conclusion.

With PPK, I find it's better than RTK simply because I don't have to worry about the real-time correction connection. As long as your base station is capturing data continuously over your point capture period you'll be able to apply PPK later. And you can even do things like use OPUS to get even more accurate data.

FWIW, We use a LIDAR with co-aligned 20MP camera to get RGB point clouds and they are great for large space explorations.

A good write up! The one thing I’ll add is that in most regions that aren’t terribly remote, you can just use a VRS RTK service, avoiding the entire base station effort. You’ll still get 1cm RMSE, and all you need is a temporary subscription and a cell modem, often built into your receiver.

Correct. There is a VRS service in the area, but for me A) they didn't make it easy to purchase service and B) they didn't have a station close enough for my preference. Also worth a mention, in Europe, there's the Centipede network, which appears to be an open effort to provide RTK access.

Hi Ben! Cool seeing you on HN!

so incredibly cool

Reply on news.ycombinator.com