The result is 100% of auth requests timeout once the login queue depth gets above a hundred or so. At that point, the users retry their login attempts, so you need to scale out fast. If you haven't tested scale out, then it's time to implement a bcrypt thread pool, or reimplement your application.
But at least the architecture I described "scales".
You do, in fact, need to scale to trivial numbers of users. You may even need to scale to a small number of users in the near future.
If you have a product that’s being deployed for a new school year, yeah you should be prepared for any one-time load for that time period.
Many products don’t have the “school year just started” spikes. But some do.
It requires careful thought, pragmatism, and business sense to balance everything and achieve the most with the available resources.
I am worried by the talk of 10k daily users and a peak of 1000TPS being too much premature optimisation. Those numbers are quite low. You should know your expected traffic patterns, add a margin of error, and stress test your system to make sure it can handle the traffic.
I disagree that self-inflicted architectural issues and personnel issues are different.
they couldn't redeploy to a high-spec VPS instead?
Instead, they celebrate "learning from running at scale" or some nonsense.