> If you haven't tested or don't know how your systems are going to react when operating for a different scale,
In my opinion, this is a pretty unreasonable expectation. Not to mention any load testing is going to be done under a certain set of assumption, a set of assumptions that is almost guaranteed to be violated by actually scaling something.
What's the most naive load test you can think of? I am going to see the maximum amount of requests my dev instance can handle. Oh i can handle 10 requests per second. If I run 24 instances on a machine, and run 100 machines, then I must be able to service 24,000 RPS.
Ok... Now explain concretely, what kind of load testing would/should have been done that would have surfaced the lack of an exponential backoff without a priori knowledge of it?
The category of things you don't know you don't know at 10x scale is large. The justification for spending time "scale testing" is going to be a hard when your manager goes to social media and sees every single person wanting backups for their texts. "Hmm should we apply engineering resources to the feature we know everyone wants, or exploring the potential land of failures that may or may not have payoff, and definitely not immediate payoff." Most of these prioritization situations are directly and solely informed via experiencing outages, just like this one.
> then don't let it grow in the first place.
This is the type of insight you get from experience, it's also the type of call that I personally think would be very hard to make. As an engineer you don't always know if an outage is going to solve itself, take 10 minutes, 30 minutes, or take 10 hours. Then communicating that to someone with the power to create that policy...
How many (social) companies do you know of that haven't had absolutely catastrophic outages/security breeches?
This is the type of outage that justifies reliability work and gets resources applied to it. This is the type of thing that results in an organization creating a somewhat rigorous incident response policy that handles the business side (keeping users informed, giving them the best experience possible). This is a growing pain of rapid maturity.
I think you can talk the way you do because you have not personally faced this type of engineering situation (a guess). Easy to make comments from the peanut gallery, but try putting yourself in the engineer's or the CEO's shoes and walking through what information you have at the time and what options you have available at the time, without the benefit of hindsight. Imagine you have a pipeline of priorities you look at every day. Is spending time wooing a senior engineer to come work for you a lower priority than having a good story for what happens when you 10x your traffic in 2 weeks? Things can only be black and white in hindsight. Their response to this is infinitely more indicative of the type of organization they are than the actual outage.