Where's the fastest place to put my server? How much does it matter?
calpaterson.com
calpaterson.com
My primary, overriding criteria in choosing locations was what would be the coolest, most interesting place to visit for installs and maintenance.
I chose Zurich and Hong Kong.
Measured results have exceeded my initial models.
(Love your work)
Though all things considered, I don't really miss sitting inside a cold, loud datacenter or standing in front of a tiny KVM monitor for hours just to load CD's to do a software upgrade.
Did you know that we have "drive up" access ?
If your account is in San Diego, you can just drive up to the datacenter and connect to our Ubiquiti LR AP. No Internet required.
We had it deployed in Denver as well but we moved datacenters a few years ago and are in a tall high rise[1] with difficult access to ground level or roof for antennae ...
[1] The "Gas and Electric" building which is the big carrier hotel in Denver ...
Of course we have switches in our racks and of course we could find a way to plug you in ...
However, there would be a lot of administrative overhead just to get you inside the datacenter in question, not to mention how we would clear you and your equipment, etc.
We regularly accept bare SATA drives as ingress and we support that process in reverse - most likely we would steer you towards that route ...
DG&E is a weird and wonderful building. Used to have a client that had a suite in there. For those unaware, it is an old, old building, right next to the building that houses the Denver main telco switch. It was built over a hundred years ago, and has been Denver's largest (? maybe that info is out of date) meet-me room for Internet providers. Being such an old building, you can imagine it has all sorts of problems retrofitting modern requirements for cabling and power generation/backup.
The issue is the he.net POP. We did use a fiber interconnect across Denver for several years to the coresite datacenter on (Denver) Pearl St. That was free because of a weird handshake deal we had from back in the day but when it stopped being free we really needed to go to DG&E to get 10gb links from Hurricane ...
I will also note that we partner with DenverOnline / Firstlink consulting which has their base inside DG&E so that also made good sense.
These are all business and personal relationships that go back to the mid-nineties and the Boulder bandwidth coop - we have a really great service and support infrastructure in Denver.
I hadn't realized that rsync.net had drive up service until this thread. Very cool service, even if we don't have it in Denver anymore.
He doesn't have to be cold! If he moved over one aisle he'd be nice and toasty!
Last time I was having to go into a server room, hot and cold aisles were still a new thing, so it was just cold everywhere.
Kept me comfortably in Gold Status with several airlines and the frequent flier miles allowed for a _lot_ of personal travel.
(While I miss that a lot, I'm super glad my business isn't reliant on international air travel being a thing these days...)
I’m genuinely interested to hear more.
Amsterdam, Frankfurt and (perhaps surprisingly) Marseilles are the places to go if you want to optimize for routing.
Equinix has datacenters, with he.net POP, in all three places. You could do a lot worse than standardizing, globally, on EQX datacenters and he.net connectivity ...
I think both Equinix locations and he.net presence are good indicators of lots of connectivity, and so both together indicates you should have access to many choices of transit (and peering). That said, I have no experience with actually obtaining connectivity at such sites; I have dreams, but no actual need ;)
Zurich is very quick and convenient to any of Chamonix (straight west) Davos/St.Moritz (south), Cervina/Zermatt (southwest) or Südtirol (southeast).
In all seriousness, everything from Geneva to Sion to Chiasso seems relatively close to me in Zurich. The intercity public transit is neither cheap given the distances nor especially fast but they're still all reachable within 3-4 hours. It's difficult to do a ski day trip to the places mentioned, admittedly, but there are still plenty of spots closer to Zurich that are suitable for that.
Sure, Flumserberg is close and there are good roads to Laax or Arosa.
I was just thinking that Grenoble, Innsbruck or even Munich have better access to skiing and probably have hosting providers, you can do day trips from all of them.
- Train to Arth Goldau, optionally visit the zoo there
- Train to Rigi Kulm (literally the top of a mountain, with some beautiful views)
- Cable car or cogwheel railway to a ferry terminal on the Vierwaldstättersee
- Ferry across the Vierwaldstättersee to Luzern
- Train from Luzern back to Zürich HB
If you have a resident buy you a day pass in advance, all of this costs 44 CHF ($44).
There's really a lot to do though. You can do a day trip to pretty much anywhere in Switzerland. I've done it to Geneva a couple of times, also to towns out in the boonies.
- job sucked - grew up in a place where the weather changes all the time, daily sun was driving me nuts - got mugged at gunpoint by the gang members (fourth was in a running car close by) who stole my bike as backpack (full of filthy underwear and socks from a 12 shift in a kitchen!) - probably an initiation - apartment was cockroach infested - the 7/11 near my place had groups who stayed outside watching you pay to see how much cash you had (not related to previous mugging) - the pollution was staggering - the nearest place up buy books was a mall 45 minutes away by bike, the library near me just had children's books and car manuals - when I biked to that bookstore, on the way home I got egged by some jerks in a car who called me various names related to what the thought my sexuality must be because I was riding a bike - and so forth
Just wasn't a good experience for me. Living as a cook, you are on the very low end of the income spectrum. If you aren't willing to have roommates, you live in terrible places. I'm sure if I were to live there with my current career, I'd like it much more. :)
;-)
1. The table's mins divide straight line distance by the speed of light. This gives you the time it takes for light to travel in a straight line from, say, London to New York. However, your "real latencies" are roundtrip ("ping") latencies. Thus, you need to multiply all the theoretical latencies by two.
2. Data does not travel through a fiber optic cable at the speed of light. This is because light actually bends around the cable when transmitted. These are called cosine losses, and mean the light travels roughly 5/3 the actual cable distance. So, multiply again by 5/3. (This is why HFT firms use microwave links for long distances.)
If you multiply the theoretical maxes by 3.33, you'll see that they're very close to the actual latencies you're observing. New York -> London becomes 62.7 ms optimal, so you're only 13% slower than the theoretical max.
Here on the west coast, I typically see within 10% of the theoretical min for data going over the Seattle -> Japan submarine cables.
Yes. Not to mention those Fibre aren't exactly a straight line. There is extra distance for layering the fibre route. 13% is very close to practical maximum.
That is why I asked [1] if we have Hollow Core Cable [2] soon where we get close to Real speed of light.
[1] https://news.ycombinator.com/item?id=26026002
[2] https://www.laserfocusworld.com/fiber-optics/article/1417001...
Do you know if anyone's considering these for consumer Internet?
If there's significant deployment of the cable in long distance networks, eventually that should trickle down to users. It would probably happen faster if there were competitive local networks, but regardless, a significant drop in latency across a country or ocean can be big enough to justify some expense.
Obviously the HFT crowd are very interested in these, but they are willing to pay the premiums. Also the next area is probably datacentres where latency is very important as well, and these fibres already provide similar losses to multi-mode fibres at ~900 nm wavelengths.
1. Submarine cables can't go in a straight line, since they've got to, you know, go down to the bottom of the ocean. (Which, you may have heard, is quite deep.) Also, a cable with a length of several thousand miles tends to have some slack.
2. Your packets may take a very curvy route from one city to another, even when they're not geographically that distant. This may be because your ISP is bad (and has poor/limited routes), geographic or geopolitical concerns, or just because of the way the Internet infrastructure is built. On the US's west coast, I often experience latencies 60%+ slower than the theoretical minimum when accessing servers in the central or eastern US. (e.g. SF -> Des Moines, IA at 70ms).
I would think a bigger factor would be that the cables (IIRC) don't go in straight lines (which you did allude to).
(Since somebody else has already mentioned the pinboardiness of these comments... https://idlewords.com/2007/04/the_alameda_weehawken_burrito_... )
[0] https://www.nytimes.com/2014/04/14/opinion/krugman-three-exp...
[1] https://www.amsterdamtrader.com/2014/09/hft-in-my-backyard.h...
Ha, it only just struck me that could have been an NSA-type routing issue!?!
More than once I have seen a traceroute that takes you from Miami, down to Argentina and then back up to Cali.
So you are still quite a bit removed from from the physical layers. Your packet will likely go through several electrical-to-optical and optical-to-electrical conversions, probably there will be some electric switches, plus multiplexers, all of which contain buffers. Then there is forward error correction in the physical layer which also requires buffers etc..
And you're obviously right that for many reasons the "straight path" might not be the path that is being taken, or even the fastest one.
Bottom line, estimating ping time from geographic distance is a very rough estimate. However, the longer the distance through an uninterrupted link (i.e. a submarine cable) the better your estimate, i.e. if you sit at in a google datacentre which is directly connected to their fibre backbone and do a ping to machine in a similar data centre on a different country you will get quite close numbers I imagine (I don't work for google). On the other hand if you sit somewhere in the mid-west at home and ping a server in e.g. NY or LA, not so much.
Round trip vs one-way
#2 is great background! But these cosine losses are I suppose not a theoretical limit but a limitation of fibre optics so I won't include that (but I will link to your comment!).
The reason that light travels slower in fibre is because the refractive index of slica is about 1.5 while it is 1 in glass (in reality it's a bit more complicated, it's the group index that counts, which is also approx. 1.5 however).
We built Fly _specifically_ so you can run servers close to your users. We do a lot for you on the network side, too, like terminate TLS in all our regions.
One thing to note, though, is that latencies between cities are surprisingly different than their theoretical max. We have apps on Fly with lots of South American users. It's frequently faster for people in Argentina to connect to Miami than it is to hit Sao Paulo or Santiago. The same goes for Africa.
And the "Asia Pacific" region is a monster. Most points of presence there are _thousands_ of miles away from each other. So we occasionally see people go from Tokyo -> LA instead of Tokyo -> Hong Kong/Singapore.
Yeah, most networks in South America don't interconnect with networks in other countries. They mostly connect in Miami, because international connectivity is difficult to arrange, and if you can only manage to connect to one other country, it needs to be the US. Africa likely goes to Europe rather than the US? But same thing, it's hard to connect, so first you have to connect to where the content is. And there's not (currently) enough network stability and capacity to just connect to your neighbors and rely on them to get you to the US/EU either.
I don't know if it's current, but in Japan there used to be two dominant networks, but they weren't both available at the same PoP. You would need a Tokyo-A pop to get to one, and a Tokyo-B pop to get to the other; and it was difficult to interconnect those pops.
We just released a preview Postgres + regional replicas. This setup keeps the postgres leader in one region and lets people add replicas in other regions they're interested in. It works really well for typical full stack apps.
We've also found a surprising number of customers who want to run in only one specific region. They don't spread apps out geographically, they just have a concentrated population of users in, say, Sydney. We're increasingly becoming "Heroku for <region>".
For a read-dominated access pattern that sounds quite interesting.
Then there are a lot of requests you can serve from read replicas or caches that can be close.
We almost shipped cockroach but it's missing some Postgres features that most full stack frameworks rely on.
https://fly.io/docs/app-guides/graphql-edge-caching-apollo/
I'm not entirely convinced this is a great idea (and apparently the example uses plain http between the graphql prox/cache and openlibrary.org - that might be considered a bug, I suppose: https://github.com/fly-apps/edge-apollo-cache/blob/master/sr... At any rate I'd assume whatever source you're proxying (eg your own openapi rest end points) - you might want ssl - or connect the db/api to fly.io via vpn a la: https://fly.io/blog/building-clusters-with-serf/ See also the linked: https://fly.io/blog/incoming-6pn-private-networks/ ).
But: while you can very easily use TLS to backhaul to an API, a more "modern" Fly.io way to solve this problem is with WireGuard gateways; it's trivial --- I'd argue, easier than configuring SSH certs --- to get a WireGuard link from your cache app on Fly back to AWS, GCP, or wherever your non-Fly legacy database lives.
I really think WireGuard is going to change a lot of the ways we design systems like this.
One of the most interesting things about wireguard to me is cheap ubiquitous application neutral secure comms.
If your queries have data dependencies, you'd get better results with a front end near your users, and a middle tier api near your database.
But, just TCP (or TLS) termination near your users and the real frontend near your database can make a surprising amount of difference.
On the other hand. There's a lot you can do to make sure things are fast from limited locations. Keep an eye on data size, make sure you're monitoring for and fixing slow queries, optimize your images, etc. If your html takes seconds to serve, it barely matters where you served it from.
Soon the fastest place is going to be the 5G Edge with products like vapor.io Kinetic Edge Colo, AWS Wavelength, Google Anthos for Telecommunications already making a push for it.
Ex: https://cloud.google.com/blog/topics/anthos/anthos-for-telec...
In systems in which there is common shared state between all participants (e.g. Fortnite), you fundamentally must have a form of centralized authority somewhere. In these cases, you do not gain very much by pushing things to the edge if most interactions require taking a round trip through the business state machine.
This realization is why I really enjoy the renewed push for server-side hosting technologies. Accept the fundamental physics constraints, bring all the state under 1 roof, and think differently about the footprint of a global-scale application. AWS solved this problem by making their regions more-or-less 100% standalone. The only major thing that really spans all regions is account/session management, but consider that there arent serious UX constraints around a login attempt taking more than 150ms.
I wonder if there's any big online game already which does match making preferring local servers. And I don't mean local as in "people registered in us-east region", but rather "there are 4 people ready to play in Chicago, spawn an instance there and connect them".
This is a flawed comparison because it looks like he's comparing ping latency (round trip time) with speed of light (one trip), so the "Slowdown factor" columns is off by a factor of two.
"If you just want to see what I did, best to read the makefile directly."
Sometimes I'm amazed by the power of a good makefile that allows replicating a perhaps fairly complex set of inter-dependent targets. I wish this approach was used more in academic research, even though fitting data analysis and modelling within a standard makefile can get tricky (e.g. some passages going through remote cluster computing, some models involving large number of files that would need to be listed as dependencies)
Splitting static content into smaller files allows for better cache utilization, but increases page load times due to higher latency when those items aren't cached. Bundling content reduces roundtrip penalties (latency, various per-request processing overheads) at the cost of greater bandwidth usage.
Wealthier users usually are latency-limited, as opposed to bandwidth-limited. Mobile users and poorer users in wealthier countries are usually limited on both. Users in poorer countries have it even worse.
The only way that you can "win" appears to be by making your content as static and as simple as possible.
Please.
One could compare CDN and HTTP server with setting TCP initial window or without.
Linux command to change initcwnd ``sudo ip route change default via 192.168.1.1 dev eth0 proto static initcwnd 10```
For more information about initcwnd see: * https://blog.imaginea.com/look-at-tcp-initcwnd-cdns/ * https://tma.ifip.org/2018/wp-content/uploads/sites/3/2018/06... * https://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum...
(which didn't seem to be worth linking to)
I.e. you can build a website that downloads some data into the browser while the visitor is viewing the first page and subsequent pages will be delivered from inside the browser without hitting the network. They may not be too happy about the extra traffic, especially when on mobile.
> Caches work great for CSS files, images and javascript - stuff that doesn't chance for each user. It doesn't work as well for the responses to API calls, for which the responses are different for each user, and sometimes, each time
Regarding this, there seems to be some people addressing the issue now (I have fly.io in mind, and maybe Vercel but I believe the latter use lambdas behind the hood so probably less effective).
For trading platforms it's fun monitoring latencies on websocket heartbeats, sometimes i get lucky and discover some platform is really nearby (e.g. 1ms to 3ms) which is very nice.
Still, nothing beats localhost as the fastest place on earth.
PDS: In other words, if you're looking for great connectivity points (in America or abroad) -- look for peering points already in use by the financial industry...
It makes a lot of sense...
Optimize for MOS ( https://en.wikipedia.org/wiki/Mean_opinion_score ) and take into account how latency affects that score.
Instead of having your users go through the public internet, they connect to some proxy which is closeby and sends the traffic through the „privat“ line.
1 Mb = 1 megabit
1 mB = 1 millibyte
1 mb = 1 millibit