Hacker News Reader

Comment by onei | Hacker News Reader

How did you get to six? Is that just sending a single packet without dealing with responses? Never thought about this before so I'm curious.

View on HN

Source container -> VM -> host -> host -> VM -> destination container

This is one of the benefits to running containers on bare metal.

It's not as bad as it sounds as a few of those transitions can be implemented via zero-copy APIs, so you're really just passing around pointers to shared memory.

The transitions happen efficiently, but the packet still has to spend cpu time traversing the network stack at all 6 hops. So the packet moves all the way through the routing / natting tables of the host, and then very quickly ends up in the VM. Then has to traverse the whole routing / natting tables of the VM.....

If there's NAT involved (as was posited in the start of the thread), then you can't just copy, you also need to modify.

Correct me if I'm wrong, but I don't think the container transition introduces any additional copies and hops. Containers are not VMs. They just use isolation that the kernel enforces. Whether you send a packet from inside or outside a container to the network should be exactly the same amount of work unless you opt-in to run some extra software-level NAT step.

You're wrong and right, because "container" is ill-defined. If you run your container in the host network namespace, then there's zero overhead. It's more common to have your container's "nic" be one end of a veth pair. That's not free.

Finally many platforms like docker, in the past, will spin up userspace proxies (docker-proxy) which is even more expensive.

You are correct that containers can be implemented that way, but docker containers aren't. They use NATing / bridging / veth pairs, which is much more usable but also much slower.

This hasn't been the case for years. See, e.g., SR-IOV, which allows modern PCIe network adapters to completely bypass a host virtual switch and present itself directly to a VM.

https://www.intel.com/content/dam/doc/application-note/pci-s... is a good overview.

Also, at least on AWS, you can directly attach VPC network interfaces to containers, which (among other things) obviates the need for software bridging, veth pairs, etc.

Reply on news.ycombinator.com