Kubernetes Networking: Behind the Scenes | Hacker News Reader

Kubernetes Networking: Behind the Scenes

itnext.io

182 pointsonig199033 comments

On the Nomad project we've taken a distinct approach to networking: treat the IP layer as an implementation detail and attempt to abstract it away. This has led to a few design decisions:

1. Global node addressing - Given a region + node ID you can query any node from any other node.

2. Global allocation (pod) addressing - Given a region + allocation ID you can interact with individual allocations and tasks (metrics, files, logs, remote exec, etc).

3. Global name-based service addressing (via Consul) - whether via Consul's DNS interface, catalog API, or Connect service mesh, services discover one another by name instead of by IP or port. The service's address (ip+port) is hopefully an opaque implementation detail that never needs to be externally observed.

With all 3 of these taken together there's no need for overlays, distinct subnets, or unique IPs-per-allocation. Datacenters may even have overlapping subnets and Consul Connect will still allow services to communicate via global name based addressing. Even without Consul, almost all Nomad commands will route to a remote region and execute on the correct node regardless of subnet overlapping.

This does cause some friction at the edge: when translating from external IP based routing to internal services (aka ingress). However, there are lots of load balancers that integrate with Consul and therefore translate between IP and name based routing adequately.

Really any service unhappy with our "route by name, not by ip" design is going to be a bit awkward to use, but routable IPs and ports are selected and exposed by Nomad's scheduler - or advertised in Consul if the address is overridden at runtime.

If getting a better understanding of exactly how Kubernetes network is of interest, I'd also recommend this episode of TGIK https://www.youtube.com/watch?v=IhbJ3ll4usI

In general actually, I've found TGIK really useful in improving my understanding of the underpinnings of k8s. They have a load of deep dive episodes that focus on a specific part of the solution and generally have a load of hands-on examples.

Shame the audio got ruined somewhere along the way.

it fixes about 2:30-3:00 in.

I feel like k8s networking is 10x more complex than it needs to be...

I think it's because they still want to use the socket API, but to "pretend" you can open a socket to a pod, or to a service, etc.

Instead, they should have defined a new API, like "connect('k8s:service:foobar?selectionstrategy=leastloaded')"

Then they could have made simple LD_PRELOAD shims to get their magic API to work with existing software without modification.

There would be no need for millions of IPtables rules, virtual IP addresses, port remapping, proxy processes, etc.

It's complex because K8S is used in a wide variety of complex environments.

Secure dynamic overlay networking, plus egress is inherently not a trivial thing. Then add customization/extendability on top.

The important thing to get is that it uses really simple building blocks, very UNIX-y. Every time a new container starts it starts a program (the CNI driver) and pipes in a JSON. And the driver does what it wants (executes iptables, BGP announces a prefix, commits something to its internal DB) and spits out some JSON that tells what IP the container got.

And .. that's it. No magic. Everything has to be configured, arguments passed to some daemon at cluster setup time, and so on. Yes, kubelet and apiserver both have hundreds of command line args, but [almost] all of them are very clearly associated with tuning a functionality (or passing some data forward to some daemon/driver/component) - and thus if you don't touch it but it works, yaay, you don't have to care about it (though of course it might turn out that you'll need to scale up/down/faster/lighter).

The socket API is fine, even with a new shiny API the network fundamentals would have to be managed just the same way. The Linux kernel would be the same, just as Ethernet, IP and so on. Reinventing all of those wheels just to provide transparent pod-to-pod networking doesn't seem to ever worth it.

How is that different from today's use of KubeDNS?

Kubernetes' networking would arguably have been less complex (multiple CIDR subnets, proxying, etc.) if they'd embraced IPv6. But Google doesn't seem big on IPv6.

API incompatibility is largely why AppEngine failed. Cloud adoption suffers greatly when the platform is not compatible with the data center.

That is basically how consul connect works: https://www.consul.io/docs/connect/native/go.html

Not as I understand it. It uses DNS resolution, so you connect to something like foo.web.service.consul. Kubernetes's KubeDNS works exactly the same way with Kubernetes service objects.

From that page:

  // Create an instance representing this service. "my-service" is the
  // name of _this_ service. The service should be cleaned up via Close.
  svc, _ := connect.NewService("my-service", client)

and when http is used:

> The hostname used in the request URL is used to identify the logical service discovery mechanism for the target. It's not actually resolved via DNS but used as a logical identifier for a Consul service discovery mechanism. It has the following specific limitations:

Ah, I see. That's presumably possible because Go has its own DNS resolver, and it (net.DefaultResolver) can be overridden by libraries; no LD_PRELOAD needed.

it's because connect.NewService is the consul client, not something like net.DialTCP

Oh, they require all TCP connections to go through that. That's a bit heavy-handed — requires rewriting all client code to adapt to that pattern. The nice thing about KubeDNS is that you don't have to. Same with Istio.

I brought up the native client because you mentioned "they should have defined a new API".

That wasn't me. I'd argue the opposite. Though I'd also maybe argue that SRV records is what we should have standardized on for this stuff a long time ago.

Integrating with app code is optional. The default is to use a sidecar.

It's surprising k8s being so new seems to have no concept of ipv6

It does, in alpha.

Soon to hit beta though: https://github.com/kubernetes/enhancements/issues/508

Most internal networks still run IPv4, and will continue to do so for a long time.

Because ipv6 is not a simple addition, but a complex suite of new protocols meant to replace v4, except that is never going to happen in our lifetimes.

v6 should've simply been v4 with more bits

IPv6 should not be an addition to Kubernetes, but the default. IPv4 could then be an addition for when you need external communication to IPv4 networks. There's no reason why IPv6 isn't the default for internal communication between containers.

We've been deploying IPv6 for at least 10 years and have hosts that are exclusively IPv6. Then along comes Docker and Kubernetes with their retro IPv4 only stack and now customer wonder why we're struggling to make containers work or suppliers that can't understand why we can't easily deploy the latest version of their software, which is only distributed as a Docker image.

Well, since k8s is developed by mere humans on simple notebooks, and initially it got deployed to stock Linux they started with v4. And they had to start with v4. And to get more contributors they also needed v4.

Yes, the network effects are strong with this one.

But soon. Bit by bit, I hope the true dualstack will come, and then the v4 will fade away over generations.

No backwards compatibility could be IPv6's biggest mistake. Now everyone is forced to maintain 2 IP addresses.

I think that’s intended as a feature. IPv4 is just not good enough, we will run out of addresses. I think they want to kill IPv4 and replace it rather than improving upon it. At some point programs will ignore IPv4 altogether and our IP addresses will not be fundamentally flawed anymore.

layers are only ever added, never removed

So far no one ever managed to come up with a plan that is backward compatible and does enough. (Aka the fundamental problem is that if you change v4 semantics you have to deploy new software/firmware/hardware, and if you are doing that anyway, a clean slate seemed like the easier option.) And don't forget v6 has well defined semantics with regards to v4 interoperability for a lot of things (::ffff:, NAT64, DNS64), so you can be on v6 and reach all of v4 via a gateway.)

that's not true. you can have IPv6 only in the backend and expose IPv4 only at the very frontend (like Cloudfare does the other way around).

Dual stack are not really needed and usually add complexity for few little benefits.

The main problem of IPv6 is the lack of people willing to learn it, teach it and adopt it.

Something brand new like Kubernetes should have from the start been IPv6 only , with IPv4 at the egress parts only.

Did anyone do some measurements on k8s/container latency overhead in networks?

Yes. And they are _really_ easy to find. https://machinezone.github.io/research/networking-solutions-... https://itnext.io/benchmark-results-of-kubernetes-network-pl...

Edit: added one more link

I'm sorry that I'm an _idiot_.

Thanks for the links.

I appreciate that you also shared Alexis Ducastel benchmark.

I love the idea of Cilium using BPF and XDP. They are the right tool for this type of software. And they extend it to do really cool things with it.

The upcoming IETF hackathon in Singapore has a project for this subject.

https://mailarchive.ietf.org/arch/msg/hackathon/PjbOkxx-5O0o...

Reply on news.ycombinator.com