Interesting. Can you give an example of why DNS based service discovery was needed?
Not to be flippant, but any case where things need to connect to other things. At scale they usually need to connect to other things through some sort of load balancer. You get that out of the box with kubernetes for services hosted inside the cluster, and there are straightforward solutions to ingress for clients outside it. Another important feature is pod scheduling. Yes you could wire up a few machines using docker compose and any of a few different networking approaches, but if one of your VMs dies are those workloads going to move to a healthy instance by themselves?
But you don't need containers either.
Service discovery just makes it easier to link up services if your architecture is truly microservice. We used to have an incredible amount of config that tightly coupled our API's.. now we use a combination of service discovery and an API gateway (Ambassador) to decouple the services, cut down on the number of random endpoints in our config, and we also get the added benefit of load balancing, rate limiting, and additional logging.
There's always a tradeoff with scale. If you have four servers than obviously all of this stuff is overkill.
I disagree, actually. I have four servers at home, and have some pods that have been running little tinkery things, and a bunch of open source software, with ridiculous uptime and little or no effort, even when I reboot one of those "servers" to do some gaming on Windows.
Now, do I need that uptime for all of those services? Not really. But for some of them, I want it, and it'd be annoying if I had to go figure out why they'd stopped running. The reality is, things just keep ticking without me worrying when they're on k8s.
These skills transfer into very in-demand job skills as well, and if I ever build anything that gains traction, I already have all the tools, configs, and knowledge to deploy that app across 500 generic cloud servers.
I wouldn't recommend Kubernetes if you only have one application you run in production. Just rsync your production image to production whenever you remember to do a release. But if you have more than 1 thing, it's time to start thinking about it, because the "do whatever" that works great with 1 thing starts to break down when 1 becomes 2. That is not Kubernetes's fault. That's just the nature of the beast.
And it just works. Simply and intuitively. With everything. Because it's been an IETF standard since 1983.