Disabling Docker ICC still allows raw ethernet communication between containers
github.com
github.com
Relevant filed issue: https://github.com/moby/moby/issues/36355
There are three immediate workarounds.
1. Dropping `CAP_SYS_RAW` in docker run will prevent the raw socket from being opened (and is always a good idea).
2. Placing each container on it's own bridge network blocks this communication but requires you to manually specify the subnets of those networks to work around dockers default limit of 31 networks.
3. Create ebtables rules, to filter below the ip layer (layer 2) see the repo for an example.
edit: Since 2004 to be precise.
https://www.freebsd.org/releases/5.3R/relnotes-amd64.html
So well documented. Gotta love it.
It sounds like the Docker folks aren't thrilled with enabling raw sockets by default, but enough people expect outbound ping to work.
(Linux's network stack does have an unprivileged ping, controlled by setting sysctl net.ipv4.ping_group_range appropriately; recent ping binaries support using it but I think most distros don't enable the sysctl. I'm guessing Docker could enable it within containers if they weren't worried about older distros that predate this support in ping.)
a) I didn't need much separation, I just wanted to run multiple environments on one box to save hardware; in this case, I enabled raw sockets, because it was convenient to ssh into the virtual environment and ping things.
b) things that I wanted really separate; for these environments, I left raw sockets disabled, the jail only contains the one executable (statically compiled); additionally, I also setup ipfw rules to prevent IP traffic from the jail from getting in or out, other than the specific things it was intended to do.
Not familiar with freeBSD jails.
Jails doesn't have any of the deployment, tools for creating images, building of images, registry for images etc. Docker also automates a lot of networking stuff away whereas with jails I usually have to use something like PF to set up port forwards; I also sometimes run into problems with the fact it shares the network stack with the host.
That being said, all the configuration is quite simple and building exactly what I need for running an application isn't too difficult. The big advantage after that being that I understand how everything is wired together and I can debug easier when something goes wrong. Doing it manually however takes up much more time.
I guess it's not clear to me if the vulnerability/bug/whatever here is "two conspiring containers can establish a covert channel" or "a malicious container can send normal-looking traffic to a container, bypassing that container's firewall rules."
It does seem like the right answer is unique bridge networks per container. On physical networks, it's hard to prevent two untrusted devices on the same L2 domain from establishing a covert channel. (And it's hard to prevent two networked untrusted, conspiring devices anywhere on the internet from establishing a covert channel, if they're trying hard enough.)
I'm afraid my initiation to ebtables (and iptables) was yesterday, so feedback such as this is highly appreciated. I will explore this and add a test to the repo.
> I guess it's not clear to me if the vulnerability/bug/whatever here is "two conspiring containers can establish a covert channel" or "a malicious container can send normal-looking traffic to a container, bypassing that container's firewall rules."
In the context of a sandbox, I purport the vulnerability is that a malicious container can bypass the firewall. The covert channel is largely inevitable because ultimately the containers can modulate cpu usage to communicate via cpu timing (I read this in a security paper somewhere, not sure how actual implementation works).
The impact of bypassing the firewall can be serious if the non-malicious container exposes any service without proper authentication (as is often done behind VPCs in the cloud).
Given that a high number of official containers contain known vulnerabilities (according to dockerhub). This is a potential issue for a large number of docker users.
> It does seem like the right answer is unique bridge networks per container.
My thinking is along the same lines, although I'm wondering if a container can communicate with the host if L2 packets may be able to traverse the network bridges. If anyone has input here, it'd be greatly appreciated.
You can use the iptables physdev extension to match based on a bridge port rather than a mac address.
So to filter between containers, you can do:
iptables -I FORWARD -m physdev --physdev-in vetha056126 -j DROP
The iptables filter table has three chains: INPUT, OUTPUT, FORWARD. In a traditional non-bridge context:
- INPUT means packets entering the host
- OUTPUT means packets leaving the host
- FORWARD means packets being forwarded through the host which are not destined for the host itself.
In the context of a bridge, FORWARD means forwarding packets from one bridge port to another. You can test the simplest case of this with:
iptables -I FORWARD -j DROP
ebtables supports «broadcast» as a destination to match ethernet frames on.
ebtables … -d broadcast
ebtables … -d ff:ff:ff:ff:ff:ff/ff:ff:ff:ff:ff:ffBoth are part of the netfilter subsystem in the linux kernel.
I have a friend that I worked with at now large company (now) with back in the startup days. Guy worked at ATT and has name on RFCs for things that make all this stuff work. We where doing the startup thing, ie the guys that understand it own it, for the company. No IT so you do it since you know it...firewall and vpn, etc (he wrote routing code, I was CSE), but startup so... It made me laugh so hard when he said “iptables makes no sense. Black magic. Let’s just use PFsense”. That had always been my view but to hear someone that wrote the stuff that makes this Internet magic work agree with me was joyful.
If your "highly secure" environment relies on docker for isolation (a software neither meant nor designed for security), then it hardly qualifies as "highly secure" in the first place.
This is just not true. Docker by default requires hardening, but container implementations are fairly secure. With Kernel/OS hardening they are quite secure.
Docker security by default is actually quite good outside of a few issues (like raw sockets), and as long as you don't mount '/' or something. Seccomp is enabled by default to mitigate attack surface on the kernel, SELinux can be enabled to further enforce access control.
Plenty of sandboxes are relying on containers now seemingly successfully. Sandstorm.io is a good example (and where I picked up a lot of tips for securing containers).
If you're convinced otherwise, feel free to share some research.
There's still some big differences between Sandstorm and Docker's sandboxes -- not as big as there used to be, but non-trivial. For example, Docker mounts /proc, albeit read-only; Sandstorm doesn't mount it at all. Docker lets the container talk to the network; Sandstorm gives it only a loopback device and forces all external communication through one inherited socket speaking Cap'n Proto. In general, Sandstorm is willing to break more things to reduce attack surface; Docker puts more priority on compatibility.
So is it "secure"? That's not a yes/no question. Security is about risk management; you can always reduce your risk further by jumping through more hoops. There will certainly be more container breakout bugs found in the future, including bugs affecting Docker and affecting Sandstorm. There will probably be more that affect Docker, because its attack surface is larger.
There will also be VM breakouts. VM breakouts appear to be less common than container breakouts, but not by as much as some people seem to think. Anyone who makes a blanket statement that VMs are secure and Docker is not does not know what they are talking about.
The only thing I'd feel comfortable saying is "totally secure" is a computer that is physically unable to receive or transmit information (airgapped, no microphone/speaker, etc.), but that's hardly useful.
In general, if you are going to run possibly-malicious code, the better way to reduce your risk is not to use a better sandbox, but to use multiple layered sandboxes. The effect is multiplicative: now the attacker must simultaneously obtain zero-days in both layers. For example, Google Chrome employs the V8 sandbox as well as a secondary container-based sandbox.
In Sandstorm's case, the "second layer" is that application packages are signed with the UI presenting to you the signer's public identity (e.g. Github profile) at install time, and the fact that most people only let relatively trustworthy users install apps on their Sandstorm servers. (For Sandstorm Oasis, the cloud version that anyone can sign up to, application code runs on physically separate machines from storage and other trusted code, for a different kind of second layer.)
I was inspired by sandstorm's supervisor.c++ while looking into container security which actually eventually led me to this issue.
> For example, Docker mounts /proc, albeit read-only;
I can't find too much information on this, does docker mount /proc from the host by default in each of the containers?
> In general, if you are going to run possibly-malicious code, the better way to reduce your risk is not to use a better sandbox, but to use multiple layered sandboxes. The effect is multiplicative: now the attacker must simultaneously obtain zero-days in both layers. For example, Google Chrome employs the V8 sandbox as well as a secondary container-based sandbox.
I think that sandboxing the language is pretty tough in the general case, since every language takes a lot of effort. In non-managed, compiled languages, it will be even tougher.
What is your opinion on tools like SELinux as a secondary layers?
I'm curious what kentonv has to say, but on modern kernels docker can make use of PID namespaces, so /proc only shows PIDs from the container's PID namespace. That said, it does still provide several information leaks like /proc/self/mountinfo showing which host directories are mounted where in the container.
In addition to PID namespaces, another isolation gotcha is users, and docker does not enable user namespaces by default since it's a relatively new kernel feature and it breaks backwards compatibility (ex: kubernetes doesn't yet support them). A good example of this in practice that many people hit issues with in the past was ulimits: if UID 1000 in a container exceeds a ulimit, it also affects UID 1000 in every other container. Docker solved this by setting ulimits to unlimited on the docker daemon process, which are then inherited by containers (this also happens to be good for performance). User namespaces are one of the big recent improvements to container security.
Sorry, I should have clarified: It's /proc for the specific PID namespace. So in theory it doesn't leak anything bad. The problem is that it's a huge attack surface; there have been bugs in /proc before.
> I think that sandboxing the language is pretty tough in the general case, since every language takes a lot of effort. In non-managed, compiled languages, it will be even tougher.
WebAssembly sandboxes non-managed compiled languages pretty well. :)
> What is your opinion on tools like SELinux as a secondary layers?
IMO it doesn't help very much, because many of the kinds of kernel bugs that allow you to escape a container tend to allow you to escape SELinux as well.
I'm wondering if containers could abuse ARP to find the neighboring addresses.
Should return mac addresses plus the associated IP address
although nmap would probably yield more interesting results.
There's a reason why filtering frames at layer 2 is a thing, but I suppose the average developer/Docker user wouldn't really be aware of it. I'm a senior network engineer at $work (ISP) and there are several locations in the network where I do this (the most common location is probably at IXP exchanges).
Layer 2 security is something that most people overlook.
The Intel Clear Containers project [0] is working on this.