Hacker News Reader

Comment by jo909 | Hacker News Reader

Could you elaborate on why containers don't perform on EC2?

I'm not running that combination myself so I wouldn't really know, but I'm not aware of problems with that specific combination or can think of anything obvious.

View on HN

I don't know about performance specifically, but you lose a lot of flexibility in managing containers when running them on the public cloud. e.g. there are rate limits so a lot of multicast protocols cannot be used as they quickly saturate that and can't be used effectively (IMO).

Containers use LXC in the Linux kernel underneath. Overhead is even lower than HVM virtualization which is a couple percent for most things. Containers are really just a smarter way of dividing resources between users on a shared linux box, something that's been going on since the dawn of time.

It might add another 1% overhead for most tasks to run containers on HVM virtual machines.

The one giant exception is network performance. The network is usually virtualized at the VM level, unless you have an "enhanced networking" VM with SR-IOV enabled. For containers it's virtualized a second time.

This makes the combo potentially terrible if you're trying to run high bandwidth stuff on low performance VM's.

I still like the combo because it allows you to give the big ole finger to AWS if they try to lock you in one day. Since your containers are isolated from the VM you can easily spin them up pretty much anywhere else.

There is a lot of incorrect information in this post.

- Containers are a combination of namespaces, cgroups, and chroot (maybe). You don't need LXC to use containers. Docker doesn't even use LXC.

- There is no overhead for running processes in containers.

- There is no requirement to virtualize networks for containers. They can be configured to use the host's network directly, at which point you are bound by the host's network capabilities. Otherwise it is typically a combination of bridges and overlay networks for which the benefits outweigh the performance concerns for most workloads.

I was wrong about docker, back when I was playing with it it did use LXC, and appears to have started out as project to make a specialized version of LXC. You're right that Docker has its own container runtime now.

The overhead for running containers is usually very low but real. The OS needs to partition low level resources that are normally shared and the scheduling introduces some overhead.

I disagree about network performance. The virtualization adds a somewhat small but non-trivial overhead here (the overhead for other stuff could probably be considered trivial)

Here is a paper I dug up on that gives results to back up my ranting. It's a bit old now but probably still holds mostly true. http://domino.research.ibm.com/library/cyberdig.nsf/papers/0...

I'd need a citation that a process running in a namespace adds overhead.

My point about network virtualization is that it is not required to use linux containers. Yes, some container tools do create network abstractions that add overhead, but they aren't required and most tools allow you to optionally bypass the abstraction and sit directly on the host's network stack.

I agree that the typical NAT or SDN setup around container networking could impact performance or at least require additional resources.

But I don't see how that would be any worse on EC2 compared to bare metal or any other hypervisor/provider.

Maybe I'm just interpreting too much into the OPs wording and he did not mean that it's a specific EC2 issue.

I've mentioned this in another comment but the short answer is: rate-limiting.

Now, Netflix, being a priority customer, may get higher limits and such. But average joe public cloud user should keep that in mind before trying to use EC2 for running containers.

Even if that claim wasn't wrong, it's an unrelated question. If there were rate-limiting problems, they'd apply to using EC2 at all even without involving containers.

I think you're ignoring the fundamental issue when deploying container based services v/s services on a multiple VM's. Usually, the architecture for containers involves spinning up a bunch of VM's and deploying some kind of layer on top of that (either K8s or Swarm or something else). When you deploy containers, they may not be on the same VM, or the overlay network itself may require some kind of communication to another container on another VM. This usually creates a lot more communication b/w hosts and rate limiting becomes the bottleneck.

Do you have any evidence of this rate-limiting showing that it's that much of a problem? People have been running clustered apps on EC2 for over a decade and it's not like you hear people saying you can't run Cassandra, ElasticSearch, etc. on EC2 because the network is limited.

Similarly, do you have any data showing that a container system has such incredible overhead compared to the actual application workload? I mean, if that was true you'd think the entire Kubernetes team would be staying up nights figuring out how to reduce overhead.

You run a compute pool, you don't spin up EC2 instances on demand for this kind of application. You scale the pool based on target utilization metrics.

AFAIK the early go versions of docker used LXC underneath, but that was ~3+ years ago

Sorry . It's hard to keep up these days, I'm 112 in Javascript years.

Wow. Your knowledge of computers is excellent for only being 4 months old.

Containers are simply processes so NOT 1% overhead! No overhead but get hard to believe.

Reply on news.ycombinator.com