Who Cares if it Scales?

I once built a system for a government client. They were terrified of it crashing under heavy load, so the contract required that the site handle a load of 400,000 simultaneous requests, and also that we demonstrate (through a 3rd party load testing company) that it could handle the load.

Building it to handle the load was actually really easy (the architecture naturally handled many simultaneous requests easily), but testing it was very expensive, because it also involved making external requests we had to pay for.

When the site finally launched, it did indeed get "a lot" of traffic. At its absolute peak, it hit around 20 requests per second.

Sigh.

I think more people should acknowledge that they're not actually going to build "the next Facebook" or "the next Amazon". Even those very, very few who are going to build the next huge site aren't going to build it that way out of the box

Designing for scalability isn't that hard, as long as you understand how big things scale and what their normal failure points are.

The biggest problem seems to be connections to the database. Front-end people use databases as a persistent store, and don't realize that databases have connection limits. Your front-end or middleware scales, but your backend won't.

Your data store should be able to handle your connection load. That might mean pulling the database information and putting it into a cache (redis, memcached).

You need to understand where the bottlenecks in your systems are/will be. Authentication is always a bottleneck; can your system handle 5 million attempts at authentication in a second? Probably not.

You also need to understand your failure modes and how to handle them. In the above case, exponential backoff won't really work. If you can only authenticate 1000 users a second the other 4,999,000 users will be hitting your service at exactly the same time, every time. Randomly distributing them across the backoff interval is pointless.

And even at your fastest throughput authenticating your users will take 5,000 seconds, or around 83 minutes. That probably won't work out. Assume you need to store tokens for all of the users, well, that means 1,000 writes to the DB a second. That probably will fail as well.

Most of the time you'll be lucky to get a few thousand users. But at some level you should understand what you'll need to do if you get lucky and your utilization pops.

The problem is that you do need to understand your end-to-end architecture to do this sort of analysis. And experience counts; if you're aware of what you need to do you can design things in such a way as to make scaling easier (ie: no major rewrite required).

I headed Engineering for a company that built a messaging application. A company acquired us and wanted to integrate our back end with their front end. This was a HUGE deal prior to acquisition and become Priority #1 after. We needed to hit a goal of roughly 300 messages per second of throughput. This was touted as being very serious for us and an all hands on deck activity. As it turns out, we already exceeded that threshold by about 20%.

The front end, however, was only capable of < 5 messages per second and since no one at BigCo anticipated their side would be the problem, they had absolutely zero anticipation of having to solve anything on their end.

The way to think about scaling is simple:

Every solution will be fine for a while, but will break at a different scale.

The goal isn't to get something that scales from the get-go, the goal is to make it so when you do need to scale you can do it without a ton of work.

Building for millions of users is expensive, time-consuming, and difficult. If you're a startup with limited funding design for a small number of users first, because you won't get millions of users. Then as your user base ramps up start upgrading your pain points.

If you do hit that millions-of-users mark immediately, hit up your backers for more money and throw hardware at it until you have time/space to rebuild your stack.

The people who care are those supporting customers who are 'too big' and engineers left trying to make it scale later, when it's falling over and no thought to scalability was included earlier because nobody cared.

Ugh. I hear this all the time. While it's somewhat correct during early stages, it can be very wrong soon afterwards. I left a company after an extremely short time because it turned out they spent a decade ignoring scalability. They ended up completely boxed in and had no clue how to fix it. After a decade of piling things on top of each other it's extremely difficult to fix without throwing large pieces out, but they didn't want to do that. We were expected to refactor the pancake stack while simultaneously building new data centers to the old design (because being able to deploy to more than one region isn't just about "SCALABILITY DUH" it's also about data laws, regulatory requirements etc and therefore selling to certain customers). And none of the infra code was written to scale, it was pretty much hard-coded to the original data center and required thousands of search-and-replace that weren't as simple as a 'sed' operation because of assumptions like "all regions have 3+ availability zones" or failure to consider globally unique naming.

The other problem with continually and habitually silencing (perfectly valid) scaling concerns is that the people who are being condescendingly told that they're "prematurely optimising" will eventually leave. When you turn this article's advice into a Harry Potter-esque spell that somebody can shout during a design session: "Premature Optimisation!" it will get abused by people who don't like detail, or want to make some stupid deadline that they agreed to before they thought deeply enough about the problem space. How long can you retain good engineers if they're constantly battling against this? When you finally saturate some choke point and your system begins to fail in unexpected ways, or your costs are too high and you suddenly realise that scalability also means scaling down too, those people who you silenced might not be there.

I was once told, while cobbling together a metrics solution spanning 8 data centers that some of the things I wanted to spend a day or two on were "premature optimisation". Until we switched it on. It was dropping 50% of metrics on the floor (useless for alerting) and we got a call from the networks team because it was saturating the links. I spend those two days on my optimisations it worked. We were extremely lucky that the breakage happened at the start, and not months later. We were lucky that it didn't just turn into a random alert for a random team in the middle of the night. We were lucky that we had a network team monitoring things adequately. We were lucky that we didn't build more things on top of the bad design that relied on it.

Sometimes one man's premature optimisation is another man's due-diligence. You need to be sure that your rewarding the right behaviours.

I don't think that there are any good answers to scaling, since it's a complex topic and finding a balance is difficult, with the situations at the both ends of the spectrum being rather bad - either crippling the future performance by unscalable architecture decisions (which may or may not ever be relevant), or not getting anything done due to overengineering ahead of time (premature optimization).

That said, personally I think that there are a few things that take very little effort and can help you avoid a "singleton" application, which cannot be reasonably run on more than one instance and thus presents not only horizontal scalability risks, but also serious risks in regards to downtime since startup might be so long that your single instance going down or needing a restart will result in a prolonged downtime.

For starters, I think that the 12 factor app principles are pretty good: https://12factor.net/

But for the most part, you just want to lean towards a "shared nothing" architecture, where you don't use local memory or local file system for persistent storage of any kind (e.g. files, session information, application caches). Sure, those are bad practices in general (e.g. all sessions being lost after an application restart if they're ever stored in memory, or even counting on a temp directory for persisting them) but I've seen them in plenty of applications in the past.

Those decisions might not be too hard to fix individually once you do care about resiliency, but if you're looking at a monolith that has been around for 5-10 years and now needs to be refactored, you're pretty much out of luck, since I've seen some with lots of tight coupling. I've actually talked about that in the past here on HN: https://news.ycombinator.com/item?id=32182047

In essence, what I'd advocate for would be doing the bare minimum towards being scalable, the things that are easy to do and also simple: like using a separate Redis instance for your caches/sessions, using S3 for storing files, always using some database for your data persistence, even if just a single instance in the beginning. Things like using feature flags in front of your scheduled processes, or enabling/disabling certain modules are underrated.

Then, if you hit a performance bottleneck 2 years down the line, you'll be able to address it relatively easily. And if you don't? You'll still have a nice architecture that should lend itself to changes well, as well as throwing out pieces and rewriting them in something else if you want (e.g. your PDF framework doesn't run on anything other than JDK8 so you need something else).

That said, if you're working on small projects that aren't meant for a business and don't exactly have customers (that expect some sort of an SLA) to worry about, then even something like SQLite being your entire persistence layer might be enough.