Ways to do load balancing wrong
queue.acm.org
queue.acm.org
A few things I'll point out in addition to this article:
[1] I've found I greatly prefer databases that handle the resiliency and horizontal scalability natively (or natively in their respective client libraries). DBs natively scaling has been one of the greatest advancements in web technologies during my career, and I'd really recommend taking a look at the technologies like Riak & Cassandra for inspiration.
[2] When scaling horizontally with load balancing, connection pooling becomes something to always pay attention to. If you have 1,000 servers, and 100 database instances, you likely don't want 100,000 active connections. Focusing on things like intra-zone preferences for servers (Server A prefers all DBs in its own rack, "zone", or "region") can be a real headache saver as you scale up.
[3] Capacity planning is always key. What are you trying to plan for? Do you want to survive an entire AWS zone outage? Do you want to be able to survive a region outage? Do you worry about single servers going down, and if something "bigger" happens you'll just eat the downtime? Make sure you and your business stakeholders agree on what "resiliency" looks like to your business. It'll help you balance cost and downtime appropriately.
[4] Remember when you're looking at spare capacity that you take into account the situation where a cache layer suffers an outage. If you have to fall back to the primary source of data, and it potentially is also suffering from degradation of service, what sort of capacity will you have?
Then there's the modern design decisions that make some of these subjects taboo, like "you should never cache anything" and "a single node can handle a million concurrent connections so why worry" and "let's make all our machines use 99% of their capacity with containers because it's more efficient and hope we can deploy standbys before the database queues too much and blows the entire stack up". With the emergence of the pseudo-Devops cargo cult, people who like using JavaScript for server backend code are designing infrastructure they've never administered before. (sorry for the rant)
There was a really great paper published a few years ago that was basically an Everything You Need To Know To Run Large Scale Enterprise Systems type paper. I can never find it when these types of threads come up, but it was dozens of pages long and really in-depth.
One thing that the book covers that I think this article glossed over was the fact that in sufficiently large systems there's never a single "load balancer" - instead there's many layers of load balancing systems at different levels of the stack. E.g.:
DNS load balancing -> high capacity network-level load balancing -> shared reverse HTTP proxy -> application server -> database (with a "load balancer" internal to the application server load balancing among DB replicas).
The SRE book seems to only have a couple sub-chapters on it, so I would recommend looking up white papers and best practices for your particular application of load balancing. Safari Books has over 17,000 matches for "load balancing" in its library, for example, and F5 has 12 white papers on it just for their products.
This was featured on Adrian Colyer's blog earlier this year (https://blog.acolyer.org/2016/09/12/on-designing-and-deployi...)
So, for example, our app servers typically were always scaled by CPU capacity. Thus, we tried to maximize the cost per compute unit (so if you were in Amazon this would correlate to one of their "c" flavors). Even if a fleet of servers was in the dozens or hundreds, we didn't usually try to use the largest flavor available. The reasoning behind this was mostly that making web apps and databases fully utilize massive machines can sometimes be a bit of a dark art. We repeatedly found a horizontally scaled fleet of "commodity" sized boxes was less of a headache than a small cluster of vertically scaled boxes.
You'll notice in my post above that I favor horizontally scalable technologies at every layer, and it's for that reason. Our database instances went wide before they went tall. If we needed more memory or disk we added a server of the same type, which adds incremental capacity. In some cases, we had emergencies which necessitated a vertical scaling event, so it was always nice having that bullet to fire when needed.
So, to wrap up my response to answer your question, no, not really. We didn't have any issues at 1000 that we really didn't also have at 500. Most of the servers we ran were no bigger or faster than what you could put in a small tower in your bedroom (mind you with Xeons instead of i7s). If I had to do it over again I'd stay middle of the pack as soon as I had enough traffic to fully balance across three servers and then scale from there.
Or not.
The parent post gives a good template for how to be constructive, if you are inspired to take that path.
The best advice I can give is to see if you can get someone from a large company that self hosts to chat with you. Not too many companies have the sort of requirements you're looking at, so it's going to be of the ilk of someone at Microsoft, Google, FB, Oracle, etc. Search your network, see if you've got a connection to someone on an SRE team there, and ask to buy them a coffee. :)
Telco is always the odd ball with redundancy. This requirement is somewhat relaxed. Generally mobile core networks are designed for 5 nines with geographical redundancy. This is all baked into the products and protocols (i.e SIGTRAN).
In this instance i'm looking at traditional HTTP for an API that interfaces to location based services.
One idea is to use DNS with the records dynamically modified based on availability monitoring and set the TTL very low. I see possible caveats with this solution though.
Another idea could be to maintain sessions across sites and have an active/active setup with round robin. Traffic is then routed to the active site. The cost here is inter site bandwidth being wasted and additional complexity being introduced.
I know of someone running their own CDN with this
I don't mean to sound harsh, but there was nothing in this article even close to novel in an academic sense, and none of it is even new field knowledge if you've spent any non-trivial time around load balancers. Perhaps my notions of what the acm publishes are wrong...
Make no mistake, it's a good article for things to look out for if you're getting into load balancing. It's just nothing new if you are already experienced in the field.
But ACM Queue it's not "a blog platform", more like a magazine. Something is new, something it's a recap, something is an opinion.
The target audience of ACM Queue the practitioner, not the researcher. My column tends to try to bring people up to a baseline of sanity rather than explore new research. 90% of the world is below (what I consider to be) the baseline for IT sanity. :-)