Container Isolation Gone Wrong
sysdig.com
sysdig.com
Well written OP.
To me, debugging like this is something that should be far more important to people than slinging words like Docker and NodeJS around all day. (and then mostly on Discord, or to them, the older Slack, but not IRC because that is too hard for that crowd -- totally unfounded opinion/rant)
I'm old enough to also be frustrated with buzzword driven development, and it's pretty annoying that so many believe Docker invented containerization, but don't throw the baby out with the bathwater. Containerization is an awesome tool and orthogonal to config management.
upcoming ESXi Update releases will no longer enable TPS between Virtual Machines by default
https://kb.vmware.com/selfservice/microsites/search.do?langu...
Sounds like a sane change to the defaults, but anyone who isn't securing against 3rd party code can turn it back on (to return to much more Docker-like security/performance).
- slow (re)start times
- greater resource consumption
Granted, "too heavy" is relative, but starting a few hundred VMs on a single host (assuming commodity hardware) is not going to work very well.
Several things:
1. We may have a different definition of "commodity hardware", but you're missing the broader point.
2. The broader point is that VMs are significantly less resource-efficient.
3. 1 & 2 notwithstanding, you're conveniently ignoring the issue of (re)start time
4. It's fine to use VMs, but it's frankly bizarre to fight tooth-and-nail over the ridiculous notion that they should always be preferred over containers.
VMs are memory intensive because they duplicate the operating system. The starting point is around 500 MB per VM. That's the only meaningful difference in resources compared to containers.
I am not discussing that they have different starting and stopping time.
And that was never the point.
For instance, the guest operations generally assume they're running on physical hardware and use spinlocks for some small critical sections. Under a physical hardware assumption this can be the correct approach, because the contending thread will leave the critical section soon and this overall performs better.
However, if the contending thread's vCPU is scheduled away by the hypervisor, the other thread may spinlock until the other vCPU gets scheduled back in. This wastes cycles.
A single operating system that uses OS-level virtualization (i.e., containers) has a more complete view of the system and can better multiplex the existing resources. That said, OS-level virtualization is generally accepted to have less isolation than VM isolation and solving the VM problems with, for instance, spinlocks might be easier than solving the isolation problems with containers, which is near intractable given the size of the kernel.
Unikernels try to take this approach and have a lot of the benefits of containers. If you squint, what we're really looking at with unikernels is a microkernel that uses virtualization support in hardware for robust process isolation. What's interesting to me is the question that I never see asked, which is whether we should revisit the microkernel architectures instead of laying more crap on top of monolithic kernels. The problems with Mach in terms of IPC time have been largely mitigated/eliminated with the L4 branch of microkernel.
Resource consumption might be more, but it's not going to be dramatically more than a container. It's not like a container uses nothing, the management and resource constrainers take up resources too.
Containers simply solve nothing and aren't 'better' in general. Containerizing certain programs might be useful, but other than that they are being hyped by the 'shiny new thing' crowd more than it deserves. On top of that, the amount of people using it vs. the amount of people that actually need it is way more of an issue than a container vs. vm debate.
The issue described in this post has nothing to do with "config management vs containers." Its' odd that this article would have "reinforced" that view. How would configuration management have prevented a noisy neighbor?
From the summary:
"The core lesson of this story: just because you are using containers and you get the impression that your applications are perfectly virtualized and isolated, don’t assume the kernel is fully isolating every underlying resource at a container granularity."
and
"Luckily, the solution is there and rather simple: make sure to deeply monitor all your applications."
That's nothing to do with any "configuration management vs containers" argument and everything to do with proper metrics collection and monitoring, which should be part of every "operational readiness" checklist whether Docker is used or not.
Lastly saying that Docker "In most cases it doesn't solve anything" is an absurd statement. Do you believe that virtualization does't solve any problems? If so why do you imagine the Linux kernel supports it?
Anyone who believes in the first place this shouldn't be running production systems...
Nowhere did I state that or even remotely suggest that containers give you the level of isolation that VMs would. My comment was refuting the OPs suggestion that "configuration management" was relevant to the article. Maybe you should go back and re-read the thread.
Part of the debugging method has to do with "let's show our product".
It doesn't seem like Kubernetes supports I/O resource limiting at this point [0][1].
In any case, after a problem like this is identified, a cluster admin can use pod affinity/anti-affinity to avoid both apps co-existing on the same node [2].
EDIT: For hypervisor-based container runtime, check Frakti (https://github.com/kubernetes/frakti)
0 - https://kubernetes.io/docs/concepts/configuration/manage-com...
1 - https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-con...
2 - http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kub...
http://www.phoronix.com/scan.php?page=article&item=linux-412...
Of course they didn't test fairness or latency, that is too hard.
Has both the discussion and the current proposed path for io separation.
I'd classify the primary root cause as a kernel bug. It's good to make use of otherwise unused memory for caches, but not to the extent that the caches grow so large they slow things down.
Secondarily, there's probably something wrong in a system where you have to constantly poll and attempt to access large numbers of files that don't exist. (But probably 100% of systems that do anything useful have at least some weird cruft like this somewhere in them at any given time, so I'm not judging.)
That's what made the article disappointing for me. Do all this impressive in-kernel debugging just to find out that you should've upgraded your systems first. Sigh...
Mind passing those resumes this way?
But say they made something wonderful, and it was cleaner and more efficient because of their use of Docker/Kubernetes, and they had taken the time to figure out the tradeoffs inherent to that approach. Is that worth penalizing, from your point of view?
Even I fall into the trap and sometimes I wish I knew about all this stuff but, alas, I prefer development.
The solution is very simple: as mentioned in the article, just use a newer kernel and always set memory limits for containers, the blog post is based on an older kernel (2.6.32) that quite a few people irresponsibly still use in containerized environments, mostly because EL6 is so popular among enterprises.
In newer kernels, allocations from object pools are now tied to the limits of the memory cgroups that requested them in userspace, if any, so you wouldn't incur in this specific issue and you would just effectively have a container not being able to use more than X MB of dcache entries (although there are probably other minor ones, for example related to sharing global kernel mutexes and such).
I had to re-read this a few times-- I think you meant EL6, right?
While it's nice to just say LOL upgrade you fool, most of us are stuck with the environment were given.
You can adjust kernel level memory behavior, in particular vfs_cache_pressure can be set very high to force dentry to empty more aggressively.
What is irresponsible is for Docker to purposefully avoid to mention that it has endless issues on these widely used OS.
The 2.6.X is used in CentOS/RHEL 6, which is the standard in numerous enterprises.
It is not a 2.6 kernel by the way, redhat is backporting tons of stuff from the 3 and 4 branches.
The first problem with this statement is the idea that there's such a thing as a "perfectly fine OS". We don't even need to consider containers, the longer an OS has been in the wild, the longer its potential vulnerabilities have been found and exploited.
Windows XP is a perfectly fine OS; using it nowadays is irresponsible.
> What is irresponsible is for Docker to purposefully avoid to mention that it has endless issues on these widely used OS.
That responsibility doesn't and should never fall on the developers of an application. The extent of one's responsibility as a developer is to define the recommendations for its use. Anything beyond that is entirely on the user.
One would go insane if one had to wonder every single operating system someone decided to use one's application in.
> It is not a 2.6 kernel by the way, redhat is backporting tons of stuff from the 3 and 4 branches.
"Backporting stuff" doesn't make it not the 2.6 Kernel, it very much is.
I challenge you to find exploitable bugs in its kernel. Windows XP is not supported anymore, while RHEL 6 is.
1. If one of the two containers caused the issue, then the why you needed both of the containers to produce the issue? Why running just the offending one was not enough?
My guess is that "worker" container requested those non-existent files from a volume mounted by the other container, is it right?
2. Kernel hash table implementation. The whole point of hash table is that it's size is O(N), where N is the number of elements it holds.
Capping the hash table size to some constant and putting all the excess elements to its linked lists makes it perform like a linked list divided by the constant, no surprise. So it sounds like there's a bug in dentry hash table implementation -- it should either increase its size accordingly to elements count, or stop accepting new/evict old entries.
Running just the offending one would have been clearly enough, since its effects would have caused the same increased latency for every other process in the system (including itself). However, using a second container to observe the performance degradation proves the point that one container is able to affect another one, which is sort of the gist of the article, since too many people think containers provide much more isolation than what in reality happens.
> My guess is that "worker" container requested those non-existent files from a volume mounted by the other container, is it right?
No, the containers didn't share any volume, the dentry cache is effectively a singleton within the kernel, so even if the set of volumes is not overlapping, all processes in the system will see a performance degradation, regardless of where the files being accessed reside.
> 2. Kernel hash table implementation. The whole point of hash table is that it's size is O(N), where N is the number of elements it holds.
Your speculation is correct, however, there are sound reasons for doing such a thing in the kernel (and not allowing the main array of the hash table dynamically expand/shrink), so I wouldn't consider it a bug per se. I'll refer you to this excellent comment: https://news.ycombinator.com/item?id=14660954
The smart solution would be to expire these objects out of the cache reasonably rapidly.
(The latency in this situation is caused not by the sheer number of entries, but by the fact that the hash table is undersized for the number of entries it gets).
In most cases memory pressure will tend to naturally limit the dentry cache size - the "perfect storm" here was almost zero memory pressure combined with a process doing a lot of negative lookups on an essentially endless list of unique filenames. For such an unusual situation, it's probably reasonable to ask the administrator to manually tune things, rather than building a more complex runtime-resizing hashtable that almost everyone won't need - especially since the failure mode is a graceful performance degradation.
All for containers but they don't solve the hard problems folks often ascribe to them, really just shows in most cases you don't need to solve the hard problems. Most of the time what containers are buying you is an easy deployment method that leverages some nice features in the OS to make believe you're on separate machines.
I'm curious what issue(s) you might be referring to here with the route cache? Could you elaborate?
25g NIC is an awful lot of 60byte packets. I'm not saying this is going to be a common concern, just that like any other shared kernel resource cgroups and namespaces aren't going to help.
0: https://www.systutorials.com/docs/linux/man/8-ip-tcp_metrics...
>"25g NIC is an awful lot of 60byte packets."
Where are you getting that 60 number from? A minimum IPv4 header is 20 bytes and a minimum TCP header is 20 bytes. Also how would a tiny TCP packet relate to the route cache? Tiny TCP packet are certainly a problem with PPS that a NIC is capable I understand that. Cheers.
That's correct, I should have included a chart explicitly measuring the I/O activity done by the two containers, but I can assure you there was literally no I/O activity, a dozen open files per second is a very negligible throughput. The bottleneck was solely in the cache.