> Professionally, my experience is that certain software components need to run together on an individual machine (e.g. database server, app server, web server), and then those machines need to be networked in a certain way (e.g. web server talks to app server, which talks to database server).
This (different workload components running on a single machine) is something that kubernetes allows you to disentangle. Kubernetes creates its own network, including cluster internal DNS. Using this you expose e.g. your app servers as a service called my-app, reachable in cluster via my-app.namespace-name.svc.cluster.local.
This targets all containers with a specific label, no matter on which node they run. Server types can be scaled independently, since chances are that the load for each does not scale the same with request volume.
Round robin for DBs does not make much sense, but there are ways to e.g. expose read endpoints with one service, and write endpoints with a different one, with open source tooling which updates the target after a failover.
Kubernetes will automatically keep your services up to date, which means if you increase the replica for e.g. app server the new container will be added as valid target, as soon as it passes ready checks, and if one container fails these checks they are temporarily removed as target. The kubernetes components will also automatically "self-heal" things like a crashed container or a failed node, by restarting the container or rescheduling the workloads on the failed node to a different one, without human intervention.
This is of course very basic, but you can finetune these by e.g. configuring that a specific server type should be spread out, i.e. that only one replica(container) should be scheduled per node, to ensure it stays available if one or more nodes go down.
Or add network policies to ensure only the app server is allowed to talk to the DB server.
If your concern is latency between app and db you can have specific config that ensures that your app server containers are only scheduled on nodes where a DB server is already running, and that the traffic from app to DB is always routed to the DB instance that is on the same node (people use similar mechanisms for cloud providers like AWS, where you want to have routing rules ensuring that traffic is always sent to targets in the same AZ, to avoid cross-AZ network charges).
These advanced examples obviously require deeper kubernetes knowledge and are not something one should just try out the first time you deploy kubernetes.
Having worked with more traditional setups I do think it is often easier to configure config like this in the standardized kubernetes API rather than in e.g. nginx config + deployment scripts + idk, systemd-unit. But this point is not "having thousand of nodes" and be half the size of google.
It also depends on your team, if you have an infra team that has a stable way to manage your VMs and apps, all the power to them, replacing them all with k8s experts sure won't give you much. It isn't easy to get an unbiased opinion about when to switch, since you need knowledge of kubernetes and your current infra to make a fair comparison, and kubernetes experts probably want to sell you kubernetes. Using a handcrafted system to distribute a lot of containers over multiple VMs to ensure HA is in general a good sign to evaluate kubernetes ;)
And being on-premise makes kubernetes attractive earlier, since the bigger cloud providers have managed solutions for a lot of things kubernetes helps you with (auto scaling, load balancing, managed container platforms like AWS ECS or Google Cloud Run)