Hacker News Reader

Comment by SamuelAdams | Hacker News Reader

Wait until they learn about Docker ignoring iptable rules.

https://www.baeldung.com/linux/docker-container-published-po...

This bit me the first time I deployed using docker.

Everything but 80/443 blocked by the firewall. "Surely it is safe to run my app server on port 8000 because no one can access that port externally."

Docker stepped in to help by making sure people could access that port.

My favourite part in Docker is what if you have 127.0.0.1 configured as your resolver (eg you are running Unbound) then Docker helpfully configure containers to use 1.1.1.1 instead.

Thanks, my ass, and give back my time trying to diagnose [a container with no diagnostic tools, because it's a lightweight container] why it can't connect to someservice.on.my.lan.

Nobody should ever mess with a system's resolver settings.

I've recently had to debug a bug with a container (probably not Docker itself) unable to reach the internet. It turns out the machine is behind a NAT64 and has IPv6 addresses only, yet something in the container helpfully reconfigures the resolver to be 8.8.8.8 which of course is unreachable.

Ubuntu on WSL is really bad about this and you have to end up using chattr to make /etc/resolv.conf immutable to prevent it from fiddling around with it. Really makes you appreciate Slackware/FreeBSD.

Running anything on NAT64 without 464XLAT is a broken setup.

I have to admit with some embarrassment that I've never deployed an ipv6-only infrastructure, so I've never had to figure this stuff out. 25+ years doing Linux/infra and it's still arcane and avoidable. I don't even know what 464XLAT is, and have to look it up.

> 464XLAT allows private IPv4 clients to connect to an IPv4 host over an IPv6 network. It only supports client-server, not peer-to-peer communication or inbound connections.

...A brief description for anyone else who didn't know.

Wouldn't 127.0.0.1 be the loopback for the container itself? So you'd need to set the DNS to the gateway IP or dynamically by resolving `host.docker.internal` (assuming `--add-host=host.docker.internal:host-gateway`)

No, that won't work if the resolver is listening on 127.0.0.1, which will most likely be the case.

Docker also has special handling for systemd-resolved: if resolv.conf points at 127.0.0.53, it will use the managed resolv.conf located in /run.

Yes.

> So you'd need to set the DNS to the gateway IP

This is the least friction way. Run your own resolver there to resolve names for containers, forward to the host's loopback for everything else.

> or dynamically by resolving `host.docker.internal`

Resolving. Name. Of the DNS server. Right?

I mean dynamically as in a startup script that resolves that hostname, then updates the resolver to the IP it returns.

Doesn't even have to be dynamic, Docker engine knows the IP of the host, so it can just directly configure `/etc/resolv.conf` with it.

Might not bind on the external interface though.

Ah.

This would require adding a script to the image itself, for each image. No go.

This is such scary stuff. I love the ease of docker (compose) but stuff like this makes me miss the old days. No surprises. Now you can deploy a huge stack of stuff but it may work differently from expectations.

Its scary stuff because the common theme of software dev/deployment is to follow common patterns without actually understanding what goes on under the hood.

And no one follows what happens under the hood because companies don't value it at all (until something goes wrong, at which point they fire the scapegoat)

That issue bit me with a dockerized mysql db. I think because I was not familiar with `ufw` at the time that after I set it up I tested to ensure you couldn't get to the db.

Surprise enused.

I agree this is surprising behavior. Other comments have placed blame at the feet of Docker, iptables, or Ufw. However the easiest workaround is to tell docker to publish the ports ONLY on 127.0.0.1, making them inaccessible from elsewhere:

- ports: 127.0.0.1:8000:8000

I have found that another long-term solution is to switch to rootless Docker[0] or Podman for local dev. That way iptables isn't mangled with because you don't elevate to a privileged user to use it. Not being privileged has some downsides, but I think the tradeoffs are worth it. Docker Desktop and Rancher Desktop also is great on Linux for this reason, although, they both require a desktop environment.

(Not disagreeing with your point -- yours is a great/quick workaround -- it just seems that many people don't know about rootless Docker)

[0] https://docs.docker.com/engine/security/rootless/

I don‘t get this. What‘s the point of publishing them on some port if you don‘t that to be.. well, public?

Docker lets you publish services for consumption by other services in your application. You don't necessarily want all of those services to be accessible to the whole world. Database, for example.

Interesting. I guess I've been lucky as I use externally managed stateful and stateless rules (aws security groups + network acls)

Developers learning how their system actually works? I'll believe it when I see it... ;)

More seriously, I believe nftables has the capacity to help here: now there can be several chains which all attach to the same hook. So docker can put rules into the FORWARD base chain of the filter table (belonging to iptables, mine is empty because I don't use iptables):

    # nft list chain ip filter FORWARD
    table ip filter {
            chain FORWARD {
                    type filter hook forward priority filter; policy accept;
            }
    }

... but then the filter_FORWARD base chain of the firewalld table (belonging to firewalld) _also_ gets to process packets. Note the numerically higher filter priority, which means that this chain runs later:

    # nft list chain inet firewalld filter_FORWARD
    table inet firewalld {
            chain filter_FORWARD {
                    type filter hook forward priority filter + 10; policy accept;
                    ct state { established, related } accept
                    ct status dnat accept
                    iifname "lo" accept
                    ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } log prefix "RFC3964_IPv4_REJECT: " reject with icmpv6 type addr-unreachable
                    jump filter_FORWARD_POLICIES_pre
                    jump filter_FORWARD_IN_ZONES
                    jump filter_FORWARD_OUT_ZONES
                    jump filter_FORWARD_POLICIES_post
                    ct state { invalid } meta pkttype host log prefix "STATE_INVALID_DROP: "
                    ct state { invalid } drop
                    meta pkttype host log prefix "FINAL_REJECT: "
                    reject with icmpx type admin-prohibited
            }
    }

Now if only tools would learn to stay in their lane, cease assuming they are the only tool managing the nftables state, we'd have ufw managing its own table, docker managing its own table, firewalld managing its own...

But docker doesn't ignore iptables, it adds iptables rule to forward packets to docker table in iptables.

Okay, docker doesn't ignore firewall rules, it bypasses them. Still annoying.

I don't think that's fair. Linux users misunderstand nftables/iptables and UFW pretends to have complete control when it really doesn't.

It's annoying, but the fault lies squarely with the lack of reporting of other chains in UFW.

If everyone is misunderstanding how something works it's because it doesn't work the right way.

Clearly people want firewall rules that take priority over everything else.

I agree, and tools that try to make something complex (nftables) simple and easy to use (UFW, Firewalld) should make that possible.

Different lists are a powerful nftables feature that is used in many system services and applications for things like routing and specialized forwarding. You can't just disable that without breaking tons of software.

Instead, tools like UFW should make it clear which ports are or aren't open and what firewall rules actually apply to the current system. They exist to save users from nftables and eBPF programming, and they should work as advertised.

Ufw advertises itself with "Ufw stands for Uncomplicated Firewall, and is program for managing a netfilter firewall" not "Ufw stands for Uncomplicated Firewall, which lets you add some firewall rules but doesn't manage all of them so you need to be extra careful if you use it."

The real problem is that iptables rules aren't composable. In this thread alone there are several tools that add iptables rules (kubernetes, docker, ufw) and they are just dumped without regards to each other.

This could be solved by more tooling.

I disagree that we need more tooling. We need better tooling. Docker and Kubernetes adding network filters makes total sense, they're useless without forwarding packets between devices.

UFW, on the other hand, pretends to make firewalls simple and easy to use when they're really not. I'm not expecting UFW to start managing Docker's/K8s'/libvirtd's firewall rules, I'm just expecting them to add a section of "ports opened by other applications" that show the open ports that UFW can't control.

Set `{"iptables": false}` in daemon.json (or pass `--iptables=false` to the daemon) and it won't do it. The defaults are convenient enshittification for your casual security-ignorant developer.

Yes but this breaks the very container routing that we need Docker for. An "all or nothing" approach is not ideal.

I'm not sure how much this routing is really needed. I use nftables instead of iptables, so I told Docker to not touch anything.

I'm gradually getting rid of Docker (NixOS containers powered by systemd-nspawn are more convenient for me at the moment), so I don't have any fancy networking for it - but I still run it. I have a bunch of static nftables rules for the bridge (`iifname "docker0" ct state established,related counter accept` etc.) so the containers can access what they need. And then Docker uses docker-proxy for the exposed ports (paired with appropriate firewall rules when it's not on `lo`, where I have an umbrella `iif "lo" accept`) and it seem to works just fine.

It does not bypasses them, it uses the firewall to add forwarding rules to the container. If you read https://www.baeldung.com/linux/docker-container-published-po... there's a bunch of ways to prevent this from happening.

By enabling port forwarding and exposing, you're essentially asking docker to configure the host for it. It's likely a surprising behaviour, but seems like ufw has a bug if you have multiple chains and groups instead of just the default one.

> It does not bypasses them, it uses the firewall to add forwarding rules to the container.

... yes, it adds a forwarding rule. Which skips over the rest of my firewall rules. One might even say that it bypasses them.

> By enabling port forwarding and exposing, you're essentially asking docker to configure the host for it.

No, I'm asking docker to listen on that port and pass it through to the container. If I tell nginx to listen on port 80, would you expect it to take that as a "do anything you can to make sure you get port 80, including rewriting the firewall tables to ignore the rule that blocks port 80"?

GP is being annoying and I agree with you for the most part and I feel Docker should have tried to play nicer with the firewall.

That said it is a mistake to consider Docker in the same class as nginx. Docker is a system for telling the Linux kernel how to set up the environments of processes. Doing whatever it can at the absolute lowest level of abstraction the kernel offers is kind of its entire gig.

The reason docker is integrating this way is to easily group the rules it introduced.

> I feel Docker should have tried to play nicer with the firewall.

The problem is on the UFW side not setting up properly when other chains are present. UFW is just a front-end for iptables, and docker integrates with iptables. Can the situation be made better? Likely on the UFW side. https://docs.docker.com/network/packet-filtering-firewalls/#...

The docker behaviour is documented...

UFW seems buggy in that it only operates on set of prefixed groups instead of looking at all the groups in iptables. Now, looking at the UFW code: https://git.launchpad.net/ufw/tree/src/backend_iptables.py?h... it seems to setup it's own chains and ignores everything else. It even filters out all other chains unless they're part of UFW.

Yeah I agree, I guess it should have been up to whoever packages Docker for Ubuntu to make sure it plays nice with the standard Ubuntu tooling. Given how Docker forward Ubuntu itself is, it probably should have made UFW aware of Docker.

Btw, ouch at that 1400 line python file..

That's a fair argument - I disagree, but it's a valid perspective and probably the root of the difference in views. I personally expect docker to be a piece that fits into the system to run containers. If docker expects to effectively be the system at least for things in its domain, then that would explain why it does things that I consider out of its scope. (There's an echo here of the argument about systemd; if you expect systemd to be everything between the kernel and userspace, then of course it includes ex. its own ntpd and cron replacement, whereas if you expect it to be a service manager then every extra function looks like an overreach)

I wouldn't expect Docker to be listening on anything. Docker isn't proxying any traffic into the container, it's just configuring how ports are forwarded from the host to the container.

its... complicated. depending on things docker might or might not be proxying the traffic. see for example https://github.com/moby/moby/issues/14856 for discussion

Fair enough, I wasn't familiar with that feature.

Seems like introducing unnecessary overhead to me, but I'm also sure there's a reason it exists, so...

> yes, it adds a forwarding rule. Which skips over the rest of my firewall rules.

... which you explicitly asked it to do by using the -p option.

Not explicitly, no. The flag is `--publish`, not `--publish-regardless-of-firewall-rules`. It looks for all the world like the usual server --listen or --port options unless you happen to know about this little "feature".

-p explicitly asks for forwarding, it doesn't ask for it to be applied before firewall rules.

The issue is UFW ignoring other firewall rules, Docker just adds itself to iptables, but UFW actively ignores other chains. The bug is on UFW being insecure by design.

No 3rd party application should be able to manipulate firewall rules. Hiding configuration details leads to mistakes.

It's a 2nd party application

For the average users, it goes over the head

Upvoted because you are correct. Calling it a bypass almost implies something more nefarious vs saying docker automatically adds its own forwarding rules.

It's not intended to be nefarious, but it can have bad consequences. I know the docker devs had good intentions to help people expose services, when they introduced this. However, they also "helped" people unwillingly expose services, oops.

Just because it was meant to be useful and harmless doesn't mean they didn't actually bypass the firewall rules by adding their own rules.

Not really. Just ufw wasn't made to deal with it.

Several containers and VMs by default just go a bit "overboard" I'd say, adding network interfaces and firewalling rules left and right. It takes people installing these for the first time by surprise.

It would be fairer to say that docker rudely stomps all over your iptables configuration. ;)

It's probably ufw putting its rules in INPUT and OUTPUT chain and docker needing them in FORWARD.

Don't blame ufw incompetence on iptables.

There is separate issue with bridging aptly described here

https://wiki.libvirt.org/Net.bridge.bridge-nf-call_and_sysct...

but IIRC docker doesn't work in that config by default.

IIRC Centos/RHEL comes with bridging iptables pre-broken because they're bad at making firewall rules there.

Not just docker, either. If you configure a NodePort on your kubernetes cluster, you'll run into exactly the same issue: ufw-sourced iptables rules are overriden by kubernetes-sourced rules.

Here's the only fix I was able to find, which doesn't seem well documented. Run

    kubectl -n kube-system edit configmap kube-proxy

and edit in this:

    nodePortAddresses:
      - 192.168.0.0/16

...where the CIDR block should correspond to the local network on your machine.

They can learn about what Docker does with iptables the traditional way, by reading Docker’s networking documentation.

https://docs.docker.com/network/packet-filtering-firewalls/

It says explicitly that docker and ufw should be considered incompatible. Docker also has a configuration key to prevent it from modifying iptables.

People only have so much time to read documentation. You can't expect them to read an entire textbook before they start using Docker.

And people wonder why we aren't taken seriously like other engineering fields.

Imagine a civil engineer telling someone they don't have time to read the documentation on the bolts their using to hold up a bridge.

Imagine a bolt seller saying you have to read this entire book to understand the bolts, instead of having a couple pages containing the important specifications.

The book _is already the important information_. This is evident that not reading it casued problems.

Then imagine getting upset because the bolts don't work as you thought they did. You go do additional research and find the information right in that book you ignored! Too late, 100,000 bolts have already been used so now we gotta deal with workarounds and other additonal problems. Maybe should have read the book.

> The book _is already the important information_. This is evident that not reading it casued problems.

I agree. The problem is that it's too much upfront information.

> Then imagine getting upset because the bolts don't work as you thought they did.

I'm more imagining that the engineer glances at the documentation and rejects the complicated bolts unless there is a really really good reason to prefer them.

Programmers, on most projects, are backed into a corner of horrific complexity. The proper comparison to engineering is refusing to do the project entirely. An engineer wouldn't be reading all these details either. So you shouldn't be using that as your factor to compare professions. It's not just "Would they read everything? Yes/No" when there's orders of magnitude difference in the amount to read.

And your average webpage isn't so important you have to refuse on ethical grounds.

This helps https://github.com/chaifeng/ufw-docker

Isn't that simply because Docker adds its own firewall rules, in order to realize its docker networks?

I think podman rootless behaves much better than docker regarding firewall "bypassing"

Reply on news.ycombinator.com