Common usage 95% of of tenants is 20% of your traffic.
1% is 60% of traffic. Rest is the Rest.
Balancing is always weird. 4000 tenants can share a 3 servers and be fine. 1 client might actually need dedicated hardware.
Turns out AWS gets painful to manage beyond 1000 tenants. You start running into various account limits.
Devops it so that you can create new tenants with a script (in fact a script that is run by the signup codebase) that starts them out in cluster A.
Migration to cluster B would definitely be involved if you're not using a shared database - so potentially be ready to know your potential clients up front to get them on the correct cluster ahead of time.
I’ve often had teams of 2-6 to handle every technical aspect of a company with millions of visits. That includes feature development and support. K8 seems to need a large team just to keep it running.
At one point we had a dozen different Amazon accounts to keep everything separate. Thankfully the business model was crap, so if resolved itself.
Did Amazon mere said, "you are doing it wrong", without an explanation? In retrospective, do you think you could have done better? I'm looking to hear the lessons you learned, so may be, I don't end up making that mistake ;)
S3, I think they did up the bucket limit on request, but maxed out at 1,000. That was years ago though.