Essentially why we made a choice to go with Swarm - less features, although all the features we needed are there (restarting containers, scaling them up or down and across several machines).
However, those bugs seem to be fixed after version 19.x, but I'm still bitter about all the problems that I had because of simple features failing on me.
Kubernetes is more of an everything-and-the-kitchen-sink approach that you can’t really outgrow but because of its monolithic design, debugging can be quite challenging. It does, however, come with a much larger and much more active community that you can go to for help.
Swarm is great to get simple orchestration going, but it doesn't really do the same thing.
With k8s you can configure multi-point routing, storage backends and load balancing among other things.
With swarm, you get ... Overlay network (which is shit if you actually need to really scale) and 'scaling', which starts n containers of the same service.
Swarm feels more like an mvp which actually works even for small teams. K8s is more like a behemoth which only dedicated personal can fully tame with a silly amount of features most people don't need.
We've used both at my current job for toy projects (self hosted). Never in production however.
And I'm a personal user of gcp - which works wonderfully .. albeit more expensive than simple root servers
Load balancing should be a solved problem already. Swarm and Kubernetes should be using dead simple off-the-shelf software for ingress and load balancing. Any competitor should be able to use the same solutions. To put it another way, this shouldn't be a differentiator.
The problem is that the functionality in tools like nginx are still tied to static network architectures that evolve slowly, and don't take advantage of things like diurnal variability in workloads.
The nginx ingress controller for example, under the hood, just generates and applies a big ol' nginx config! You can extract it and view it yourself, or extend it with a Lua script if you want to be fancy and do some FFI inline with processing the request, etc.
Beyond that there's a sea of options for more native integrations that will depend on whether your LB vendor has an K8s integration, how friendly your networking team is, and how much code you're willing to write.
Lots of people have different definitions of 'easy' which is why I didn't say 'easy'. But how did you get this far off the rails with 'dead simple'?
I learned the hard way that GKE, in using GCP's load balancers don't support the same syntax for Ingress patterns as when you use an nginx Ingress. Definitely read the documentation thoroughly!