Can anyone share practical examples or stories about working with large clusters?
How do you decide when to partition k8s workloads across clusters? Are there general rules/heuristics that are helpful?
Some common top-level patterns I've seen:
1. A politically powerful central platform team builds & operates the whole infra foundation for developers. This tends to lead to fewer cloud accounts, and large shared clusters (it can be done in a multi-account, multi-cluster kind of way though).
2. Less powerful cloud engineering team(s) that do guard-rails, interconnect, patterns etc, but leave product engineering teams to manage the actual workloads. Typically results in "cluster per team or project", ideally stamped out from some shared IaC.
3. Various forms of anarchy (I don't mean this pejoratively) that result from widely distributed authority. Product engineering teams are powerful enough that no central team can really "tell them what to do". This tends to result in multiple clusters that are built and run in different ways.
Very large organisations can have pockets of all 3 models.
From my perspective the strongest forces that "push" towards big, centralised clusters are:
1. Powerful central teams that also operate. They're OK with trading off fewer things to look after for potentially larger blast radius. Also the capacity to set up the more complex security controls required for multiple teams and projects to co-exist in the same cluster.
2. Large homogenous workloads are a natural fit for big clusters. Homogenous workloads tend to have 1 "entity" looking after them, so no matter what organisation model, you can end up with big clusters.
3. Tight coupling between workload components. If you have a lot of different things that all need to talk to each other quickly and cheaply you will want to put them close together if you can.