Hacker News Reader

Comment by recuter | Hacker News Reader

> Even on this smaller extract, these services already use a large portion of my 1TB SSD (667 GB total). Assuming the usage scales proportionally, I would need around 3.7TB of storage for the entire planet. Not to mention the RAM requirements also scaling.

Why? 20TB consumer storage is a thing. https://www.amazon.com/Western-Digital-20TB-Internal-Drive/d... ($420, heh)

What in the implementation makes the RAM usage explode?

View on HN

Serving large data sets at acceptable latency typically requires some proportional amount of RAM to keep frequently accessed files and indexes in memory (especially if you're serving off of HDDs).

It depends on the service, but we frequently find geo services are RAM limited up to a certain point (at which point there's enough cached to make it CPU limited).

I'm curios about self hosting for personal usage. An offline google maps, just on my laptop, or hosted somewhere but serving very few users.

If it's just local--depending on the exact service--you probably wouldn't need a lot of RAM. Expect memory requirements to scale somewhat sub-linearly with diversity of regions simultaneously accessed.

Valhalla in particular in an in-memory router, so it's holding the whole transportation graph in memory. Not sure about the tiling and geocoding pieces. Most organizations with a "need to serve the world" requirement presumably don't balk at buying all-the-RAMs. Amazingly, the max-RAM-in-machine number has grown faster than the data-in-the-world number, and just being memory bound is a perfectly reasonable design decision, given the performance and complexity wins.

Osrm is in memory as well. We were serveing most of Europe and Asia from a box with 128 Gb RAM. It can use a mmaped file but it's too slow, even on a NVME drive. If you route anything more than 300 kms it will take a few seconds and then timeout.

Reply on news.ycombinator.com