DHCP is not blocked by ufw/iptables | Hacker News Reader

DHCP is not blocked by ufw/iptables

unix.stackexchange.com

182 pointstimost123 comments

Wait until they learn about Docker ignoring iptable rules.

https://www.baeldung.com/linux/docker-container-published-po...

This bit me the first time I deployed using docker.

Everything but 80/443 blocked by the firewall. "Surely it is safe to run my app server on port 8000 because no one can access that port externally."

Docker stepped in to help by making sure people could access that port.

My favourite part in Docker is what if you have 127.0.0.1 configured as your resolver (eg you are running Unbound) then Docker helpfully configure containers to use 1.1.1.1 instead.

Thanks, my ass, and give back my time trying to diagnose [a container with no diagnostic tools, because it's a lightweight container] why it can't connect to someservice.on.my.lan.

Nobody should ever mess with a system's resolver settings.

I've recently had to debug a bug with a container (probably not Docker itself) unable to reach the internet. It turns out the machine is behind a NAT64 and has IPv6 addresses only, yet something in the container helpfully reconfigures the resolver to be 8.8.8.8 which of course is unreachable.

Ubuntu on WSL is really bad about this and you have to end up using chattr to make /etc/resolv.conf immutable to prevent it from fiddling around with it. Really makes you appreciate Slackware/FreeBSD.

Running anything on NAT64 without 464XLAT is a broken setup.

I have to admit with some embarrassment that I've never deployed an ipv6-only infrastructure, so I've never had to figure this stuff out. 25+ years doing Linux/infra and it's still arcane and avoidable. I don't even know what 464XLAT is, and have to look it up.

> 464XLAT allows private IPv4 clients to connect to an IPv4 host over an IPv6 network. It only supports client-server, not peer-to-peer communication or inbound connections.

...A brief description for anyone else who didn't know.

Wouldn't 127.0.0.1 be the loopback for the container itself? So you'd need to set the DNS to the gateway IP or dynamically by resolving `host.docker.internal` (assuming `--add-host=host.docker.internal:host-gateway`)

No, that won't work if the resolver is listening on 127.0.0.1, which will most likely be the case.

Docker also has special handling for systemd-resolved: if resolv.conf points at 127.0.0.53, it will use the managed resolv.conf located in /run.

Yes.

> So you'd need to set the DNS to the gateway IP

This is the least friction way. Run your own resolver there to resolve names for containers, forward to the host's loopback for everything else.

> or dynamically by resolving `host.docker.internal`

Resolving. Name. Of the DNS server. Right?

I mean dynamically as in a startup script that resolves that hostname, then updates the resolver to the IP it returns.

Doesn't even have to be dynamic, Docker engine knows the IP of the host, so it can just directly configure `/etc/resolv.conf` with it.

Might not bind on the external interface though.

Ah.

This would require adding a script to the image itself, for each image. No go.

This is such scary stuff. I love the ease of docker (compose) but stuff like this makes me miss the old days. No surprises. Now you can deploy a huge stack of stuff but it may work differently from expectations.

Its scary stuff because the common theme of software dev/deployment is to follow common patterns without actually understanding what goes on under the hood.

And no one follows what happens under the hood because companies don't value it at all (until something goes wrong, at which point they fire the scapegoat)

That issue bit me with a dockerized mysql db. I think because I was not familiar with `ufw` at the time that after I set it up I tested to ensure you couldn't get to the db.

Surprise enused.

I agree this is surprising behavior. Other comments have placed blame at the feet of Docker, iptables, or Ufw. However the easiest workaround is to tell docker to publish the ports ONLY on 127.0.0.1, making them inaccessible from elsewhere:

- ports: 127.0.0.1:8000:8000

I have found that another long-term solution is to switch to rootless Docker[0] or Podman for local dev. That way iptables isn't mangled with because you don't elevate to a privileged user to use it. Not being privileged has some downsides, but I think the tradeoffs are worth it. Docker Desktop and Rancher Desktop also is great on Linux for this reason, although, they both require a desktop environment.

(Not disagreeing with your point -- yours is a great/quick workaround -- it just seems that many people don't know about rootless Docker)

[0] https://docs.docker.com/engine/security/rootless/

I don‘t get this. What‘s the point of publishing them on some port if you don‘t that to be.. well, public?

Docker lets you publish services for consumption by other services in your application. You don't necessarily want all of those services to be accessible to the whole world. Database, for example.

Interesting. I guess I've been lucky as I use externally managed stateful and stateless rules (aws security groups + network acls)

Developers learning how their system actually works? I'll believe it when I see it... ;)

More seriously, I believe nftables has the capacity to help here: now there can be several chains which all attach to the same hook. So docker can put rules into the FORWARD base chain of the filter table (belonging to iptables, mine is empty because I don't use iptables):

    # nft list chain ip filter FORWARD
    table ip filter {
            chain FORWARD {
                    type filter hook forward priority filter; policy accept;
            }
    }

... but then the filter_FORWARD base chain of the firewalld table (belonging to firewalld) _also_ gets to process packets. Note the numerically higher filter priority, which means that this chain runs later:

    # nft list chain inet firewalld filter_FORWARD
    table inet firewalld {
            chain filter_FORWARD {
                    type filter hook forward priority filter + 10; policy accept;
                    ct state { established, related } accept
                    ct status dnat accept
                    iifname "lo" accept
                    ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } log prefix "RFC3964_IPv4_REJECT: " reject with icmpv6 type addr-unreachable
                    jump filter_FORWARD_POLICIES_pre
                    jump filter_FORWARD_IN_ZONES
                    jump filter_FORWARD_OUT_ZONES
                    jump filter_FORWARD_POLICIES_post
                    ct state { invalid } meta pkttype host log prefix "STATE_INVALID_DROP: "
                    ct state { invalid } drop
                    meta pkttype host log prefix "FINAL_REJECT: "
                    reject with icmpx type admin-prohibited
            }
    }

Now if only tools would learn to stay in their lane, cease assuming they are the only tool managing the nftables state, we'd have ufw managing its own table, docker managing its own table, firewalld managing its own...

But docker doesn't ignore iptables, it adds iptables rule to forward packets to docker table in iptables.

Okay, docker doesn't ignore firewall rules, it bypasses them. Still annoying.

I don't think that's fair. Linux users misunderstand nftables/iptables and UFW pretends to have complete control when it really doesn't.

It's annoying, but the fault lies squarely with the lack of reporting of other chains in UFW.

If everyone is misunderstanding how something works it's because it doesn't work the right way.

Clearly people want firewall rules that take priority over everything else.

I agree, and tools that try to make something complex (nftables) simple and easy to use (UFW, Firewalld) should make that possible.

Different lists are a powerful nftables feature that is used in many system services and applications for things like routing and specialized forwarding. You can't just disable that without breaking tons of software.

Instead, tools like UFW should make it clear which ports are or aren't open and what firewall rules actually apply to the current system. They exist to save users from nftables and eBPF programming, and they should work as advertised.

Ufw advertises itself with "Ufw stands for Uncomplicated Firewall, and is program for managing a netfilter firewall" not "Ufw stands for Uncomplicated Firewall, which lets you add some firewall rules but doesn't manage all of them so you need to be extra careful if you use it."

The real problem is that iptables rules aren't composable. In this thread alone there are several tools that add iptables rules (kubernetes, docker, ufw) and they are just dumped without regards to each other.

This could be solved by more tooling.

I disagree that we need more tooling. We need better tooling. Docker and Kubernetes adding network filters makes total sense, they're useless without forwarding packets between devices.

UFW, on the other hand, pretends to make firewalls simple and easy to use when they're really not. I'm not expecting UFW to start managing Docker's/K8s'/libvirtd's firewall rules, I'm just expecting them to add a section of "ports opened by other applications" that show the open ports that UFW can't control.

Set `{"iptables": false}` in daemon.json (or pass `--iptables=false` to the daemon) and it won't do it. The defaults are convenient enshittification for your casual security-ignorant developer.

Yes but this breaks the very container routing that we need Docker for. An "all or nothing" approach is not ideal.

I'm not sure how much this routing is really needed. I use nftables instead of iptables, so I told Docker to not touch anything.

I'm gradually getting rid of Docker (NixOS containers powered by systemd-nspawn are more convenient for me at the moment), so I don't have any fancy networking for it - but I still run it. I have a bunch of static nftables rules for the bridge (`iifname "docker0" ct state established,related counter accept` etc.) so the containers can access what they need. And then Docker uses docker-proxy for the exposed ports (paired with appropriate firewall rules when it's not on `lo`, where I have an umbrella `iif "lo" accept`) and it seem to works just fine.

It does not bypasses them, it uses the firewall to add forwarding rules to the container. If you read https://www.baeldung.com/linux/docker-container-published-po... there's a bunch of ways to prevent this from happening.

By enabling port forwarding and exposing, you're essentially asking docker to configure the host for it. It's likely a surprising behaviour, but seems like ufw has a bug if you have multiple chains and groups instead of just the default one.

> It does not bypasses them, it uses the firewall to add forwarding rules to the container.

... yes, it adds a forwarding rule. Which skips over the rest of my firewall rules. One might even say that it bypasses them.

> By enabling port forwarding and exposing, you're essentially asking docker to configure the host for it.

No, I'm asking docker to listen on that port and pass it through to the container. If I tell nginx to listen on port 80, would you expect it to take that as a "do anything you can to make sure you get port 80, including rewriting the firewall tables to ignore the rule that blocks port 80"?

GP is being annoying and I agree with you for the most part and I feel Docker should have tried to play nicer with the firewall.

That said it is a mistake to consider Docker in the same class as nginx. Docker is a system for telling the Linux kernel how to set up the environments of processes. Doing whatever it can at the absolute lowest level of abstraction the kernel offers is kind of its entire gig.

The reason docker is integrating this way is to easily group the rules it introduced.

> I feel Docker should have tried to play nicer with the firewall.

The problem is on the UFW side not setting up properly when other chains are present. UFW is just a front-end for iptables, and docker integrates with iptables. Can the situation be made better? Likely on the UFW side. https://docs.docker.com/network/packet-filtering-firewalls/#...

The docker behaviour is documented...

UFW seems buggy in that it only operates on set of prefixed groups instead of looking at all the groups in iptables. Now, looking at the UFW code: https://git.launchpad.net/ufw/tree/src/backend_iptables.py?h... it seems to setup it's own chains and ignores everything else. It even filters out all other chains unless they're part of UFW.

Yeah I agree, I guess it should have been up to whoever packages Docker for Ubuntu to make sure it plays nice with the standard Ubuntu tooling. Given how Docker forward Ubuntu itself is, it probably should have made UFW aware of Docker.

Btw, ouch at that 1400 line python file..

That's a fair argument - I disagree, but it's a valid perspective and probably the root of the difference in views. I personally expect docker to be a piece that fits into the system to run containers. If docker expects to effectively be the system at least for things in its domain, then that would explain why it does things that I consider out of its scope. (There's an echo here of the argument about systemd; if you expect systemd to be everything between the kernel and userspace, then of course it includes ex. its own ntpd and cron replacement, whereas if you expect it to be a service manager then every extra function looks like an overreach)

I wouldn't expect Docker to be listening on anything. Docker isn't proxying any traffic into the container, it's just configuring how ports are forwarded from the host to the container.

its... complicated. depending on things docker might or might not be proxying the traffic. see for example https://github.com/moby/moby/issues/14856 for discussion

Fair enough, I wasn't familiar with that feature.

Seems like introducing unnecessary overhead to me, but I'm also sure there's a reason it exists, so...

> yes, it adds a forwarding rule. Which skips over the rest of my firewall rules.

... which you explicitly asked it to do by using the -p option.

Not explicitly, no. The flag is `--publish`, not `--publish-regardless-of-firewall-rules`. It looks for all the world like the usual server --listen or --port options unless you happen to know about this little "feature".

-p explicitly asks for forwarding, it doesn't ask for it to be applied before firewall rules.

The issue is UFW ignoring other firewall rules, Docker just adds itself to iptables, but UFW actively ignores other chains. The bug is on UFW being insecure by design.

No 3rd party application should be able to manipulate firewall rules. Hiding configuration details leads to mistakes.

It's a 2nd party application

For the average users, it goes over the head

Upvoted because you are correct. Calling it a bypass almost implies something more nefarious vs saying docker automatically adds its own forwarding rules.

It's not intended to be nefarious, but it can have bad consequences. I know the docker devs had good intentions to help people expose services, when they introduced this. However, they also "helped" people unwillingly expose services, oops.

Just because it was meant to be useful and harmless doesn't mean they didn't actually bypass the firewall rules by adding their own rules.

Not really. Just ufw wasn't made to deal with it.

Several containers and VMs by default just go a bit "overboard" I'd say, adding network interfaces and firewalling rules left and right. It takes people installing these for the first time by surprise.

It would be fairer to say that docker rudely stomps all over your iptables configuration. ;)

It's probably ufw putting its rules in INPUT and OUTPUT chain and docker needing them in FORWARD.

Don't blame ufw incompetence on iptables.

There is separate issue with bridging aptly described here

https://wiki.libvirt.org/Net.bridge.bridge-nf-call_and_sysct...

but IIRC docker doesn't work in that config by default.

IIRC Centos/RHEL comes with bridging iptables pre-broken because they're bad at making firewall rules there.

Not just docker, either. If you configure a NodePort on your kubernetes cluster, you'll run into exactly the same issue: ufw-sourced iptables rules are overriden by kubernetes-sourced rules.

Here's the only fix I was able to find, which doesn't seem well documented. Run

    kubectl -n kube-system edit configmap kube-proxy

and edit in this:

    nodePortAddresses:
      - 192.168.0.0/16

...where the CIDR block should correspond to the local network on your machine.

They can learn about what Docker does with iptables the traditional way, by reading Docker’s networking documentation.

https://docs.docker.com/network/packet-filtering-firewalls/

It says explicitly that docker and ufw should be considered incompatible. Docker also has a configuration key to prevent it from modifying iptables.

People only have so much time to read documentation. You can't expect them to read an entire textbook before they start using Docker.

And people wonder why we aren't taken seriously like other engineering fields.

Imagine a civil engineer telling someone they don't have time to read the documentation on the bolts their using to hold up a bridge.

Imagine a bolt seller saying you have to read this entire book to understand the bolts, instead of having a couple pages containing the important specifications.

The book _is already the important information_. This is evident that not reading it casued problems.

Then imagine getting upset because the bolts don't work as you thought they did. You go do additional research and find the information right in that book you ignored! Too late, 100,000 bolts have already been used so now we gotta deal with workarounds and other additonal problems. Maybe should have read the book.

> The book _is already the important information_. This is evident that not reading it casued problems.

I agree. The problem is that it's too much upfront information.

> Then imagine getting upset because the bolts don't work as you thought they did.

I'm more imagining that the engineer glances at the documentation and rejects the complicated bolts unless there is a really really good reason to prefer them.

Programmers, on most projects, are backed into a corner of horrific complexity. The proper comparison to engineering is refusing to do the project entirely. An engineer wouldn't be reading all these details either. So you shouldn't be using that as your factor to compare professions. It's not just "Would they read everything? Yes/No" when there's orders of magnitude difference in the amount to read.

And your average webpage isn't so important you have to refuse on ethical grounds.

This helps https://github.com/chaifeng/ufw-docker

Isn't that simply because Docker adds its own firewall rules, in order to realize its docker networks?

I think podman rootless behaves much better than docker regarding firewall "bypassing"

Can anyone recommend a decent book on linux firewalls, iptables and the like? Every time I wade into this I feel I'm missing to much base knowledge to make good decisions.

A classic on *nix networking fundamentals: https://beej.us/guide/bgnet/

It includes a further reading list: https://beej.us/guide/bgnet/html/split/more-references.html#...

These don't cover iptables and other firewalls themselves, but they give you enough knowledge that you can read the iptables manpage and other manuals and understand them.

Do you specifically want to learn networking for Linux? If not, the Network+ and/or CCNA certifications are a great place to start for generic network education.

I was about to ask a similar question.

I don't know much about Linux networking. But recently I dove into it a little to let my laptop connect to my desktop via Ethernet (when present), but use Wifi for everything else.

I got it working-ish using "nmtui". But I'm left pretty confused about the relationship between all the network-related tools / services / files.

E.g., is "nmtui" just a convenience wrapper around thinks like iptables and resolved? Does it work around them? Which tools are mean to be used together, vs. which ones are redundant / incompatible? And then there's systemd as well.

nmtui is a terminal user interface (tui) to control Network Manager (nm). Network Manager is a complete solution for networking that takes care of all the individual parts of networking. But as far as I know it does not change your firewall rules. You would need to use iptables for that directly, because this is not something that can be reasonably automated without actually knowing what the user wants.

systemd is an init system with a lot of bells and whistles. One of the whistles is systemd-networkd, which is also a complete networking solution, but it also delegates some work to systemd-resolved. It also doesn't take care of firewalling.

If you want to get a better understanding on how networks are set up, then you should rather look for more low-level tools like iproute2 (which provides the "ip" tool) and dhclient/dhcpd. For wireless you should look at wpa_supplicant (which is afaik used by Network Manager under the hood). And then there is of course iptables which is used to manipulate the flow of packages through the network. It should be noted that eBPF is intended to replace iptables but knowledge of iptables will still prove useful for many years to come.

nmtui is just a console UI for NetworkManager. Similar UI is most likely provided by your desktop environment of choice as well. It has nothing to do with iptables.

Thanks, that's helpful.

Any suggestion for something explaining how all these tools and service fit together (or don't)?

For a complete but not up to date guide have a look at https://tldp.org/LDP/nag2/index.html . And for more up-to-date but less complete guide https://wiki.archlinux.org/title/Network_configuration .

"Not up to date" is putting it mildly, idk if there is anything in nag2 that's still relevant these days; I get the feeling that it will just confuse more than inform at this point unless you already know the stuff and history.

Not sure what exactly are you asking about. iptables/nftables is an interface for in-kernel packet filters. NetworkManager... well, manages your interfaces and connections. systemd-resolved/dnsmasq handle DNS. They mostly just do their own thing, with NM sometimes using the other ones as needed (for example, it will set forwarding rules up and run dnsmasq when you choose a "Shared to other computers" option in NM).

https://www.dbooks.org/computer-networks-0128182008/

Linux Firewalls: Enhancing Security with Nftables and Beyond

It’s these kinds of things that makes me realize I don’t really know what I’m doing regarding networks. I would never have imagined.

Even FreeBSD’s stack, which was always much more straightforward to me, behaves like this, it seems. There’s no hope.

It's one of those things where the more I learn just makes me better realize how little I understand.

Can you imagine trying to debug this? I pity the poor soul. Hopefully all this debate will raise it to the heights of Google search results.

FWIW, this is the same behavior on OpenBSD—DHCP listens directly on bpf, which sees traffic before the packet filter.

That is a surprising revalation; that iptables filters traffic depending on Linux implementations details. One could imagine the outcry if firewall vendor X suffered a similar “feature”. Or is this well known for Linux iptable users?

If you used Linux like you would a true firewall and filtered a bridged or routed packet it should filter fine. It's really an interaction of giving low level network access to an application on the same box you're trying to do high level filtering on, then being surprised the high level filter misses the low level data.

I've never liked the way packet sockets are exposed on any operating system though. Exposing them is such an afterthought that the only way to use them is to basically act like the rest of the networking system plain doesn't exist. I shouldn't have to have raw network permissions to send and receive any packet just to be able to mark that I want to send and receive e.g. LLDP (or, on Windows, make a driver that even allows me a way to send such packets from user space in the first place). Operating systems truly offer "TCP/IP" (and UDP and maybe a few other select protocols, depending what you load) stacks not "network stacks" which give you access to each piece equally. Even plain raw IP sockets are increasingly ignored.

/rant of a network guy.

> If you used Linux like you would a true firewall

Host-based firewalls are, indeed, fake firewalls, and it's unfortunate that they chose to use the same term for functions that superficially look like they do the same thing, but are way less effective, and encourage bad network design.

The classic architecture for security was: edge router -> firewall -> DMZ -> bastion -> DMZ -> office router

You could put more firewalls, and switches and stuff, in there, but it is really important to have a bastion host as a chokepoint in the DMZ. You could have an IPS doing this instead. Or you could have a proxy server such as SOCKS.

But your firewall is pretty far away from any host that could get bright ideas about messing with it. (Because it is possible for client hosts to mess with router "firewall" settings via UPnP.)

This collapsing of topology also happens with other stuff, like NIDS/HIDS intrusion detection. Yeah, IDS can be more effective in certain ways when it has access to host-based logs and resources. But it's different than an isolated IDS listening on a TAP/SPAN where nobody logs into it and doomscrolls Facebook.

What is the DMZ, physically? I've always heard of putting your bastion in a DMZ but is that a conceptual layer or some sort of physical box sitting between the firewall and the bastion, and the bastion and the office router?

A DMZ is a buffer zone network. The DMZ is intended to be physically protected, and sparsely populated, and only by security-hardened systems with no user accounts and no data of consequence.

Acting like the whole networking stack doesn't exist is the whole purpose of packet sockets, because sometimes you want to override the things the networking stack does.

You need privileges to open a raw socket (or a packet socket, or anything else that would let you program at the IP layer) because otherwise unprivileged processes could hijack traffic for other applications on the system.

Packet sockets as-is are a great option if that's your use case but if I just want to bind to ethertype 0x88CC I shouldn't need to ask for the same permissions I can use to bind to 0x0800 or to capture all traffic. Sure, that's how things like AF_PACKET are implemented but there is nothing about being lower level that requires this all or nothing game of read/write-full-access-to-all-packets-ignoring-the-rest-of-the-os-stack-state vs bind-a-single-TCP/UDP-port. Just look at the way shared binds and privileged ports are handled for TCP, there is no reason the same concepts couldn't be applied to lower layers for more granularity.

All of that aside, none of this would explain why I need to write a custom network driver in Windows to send more than a couple predefined ethertypes from user space at all or why, in Linux, the network filter layer can't apply to any family used in the sendto() syscall instead of just sendto() calls with AF_INET set. These kinds of decisions aren't rooted in a limited scope of what packet sockets can be for or how they must interact with the rest of the network stack, it's just how they are currently built and exposed. This is great if your use case is to ignore the OS stack, but that doesn't mean doing so is the only conceivable way of packet sockets being built.

If BPF were treated slightly differently in terms of how/when certain abilities were exposed it could be a great answer to all of this. As of right now, it's just a better way to ignore the OS network stack even more than normal though.

I think your problem isn't so much packet sockets or BPF, but how bad direct socket support is for things like ICMP or DHCP or ARP.

What is a "direct" socket in this context? If it's something that gives me a restricted packet socket context from a user space context then sure, call it what you'd like as long as it does that. If it's not that then I'm not sure it really helps. E.g. my last problem was with implementing 802.1Qcj draft extensions in a cross platform daemon, so some predefined concept about how to send/receive anything more than that EtherType with custom payloads doesn't really help (particularly if it lets other things bind to that EtherType at the same time, as LLDP is a "what the last packet said" protocol).

It's a similar story in the ARP use case you mention when maybe you want to e.g. anonymously probe if a VIP is actively in use by using a 0.0.0.0 source (useful for redundant responder setups). DHCP and ICMP I could see the case for similar improvements to the RAW IP sockets (AF_INET in Linux) type as packet sockets (e.g. ability to request and permit limited scope sockets or the other enhancements mentioned above). Just that is still too limited though as the world has more than IP protocols, e.g. AF_INET gets you OSPF but not ISIS.

The idea that "no, you can't possibly mean you'd like to send Ethernet packets while still caring about what the rest of the OS network stack is doing as if you mean to just extend it instead of take its place" is precisely the issue I've found myself annoyed with. No, I don't want to use DPDK to make a custom IP forwarding stack, I just want to be registered as the app to send and receive a certain EtherType without saying I want full rights to all packets.

Sorry, I could have spoken more precisely here. What I'm saying is that it's very annoying that there is first-class socket support for TCP and UDP, but not for ICMP; there isn't a set of sockopts and sockaddrs you can plug in to send and receive specific patterns of ICMP, or OSPF, or DHCP. There could be, but there aren't, and so to do those things you generally bypass large chunks of the stack.

On the other hand, you might take away from this that the near universal limitation on first-class programmatic access to raw Ethernet (or even raw IP) has constrained the evolution of the protocols themselves --- if it were easier to code directly to IP rather than to UDP or TCP, we might have a greater diversity of IP protocols. And that might not be a good thing! We might have benefited from how clunky the socket interface is. :)

Iptables (aka linux's netfilter) processes DHCP packets like any other packets.

The ISC DHCP server though listens for raw packets and thus completely by-pass the netfilter rules.

This is similar to how you can use wireshark to see raw packets received on the physical port, before any filtering.

Any Linux process running with CAP_NET_RAW can by-pass the firewall in such a way, this includes your typical DHCP server running as root.

The question then should be why is ISC DHCP server using raw sockets? That is probably because DHCP sits in-between OSI layers, it bridges the gap between the mac address world and IP address world.

I'm not sure of the exact technical reason though. The linked SO answers talk about some case where NAT rules could be altering packets, not sure how common NAT+DHCP is used...

In the case of a DHCP client, you do need raw sockets because the Linux IP layer will not let a normal socket send packets with a NULL IP source address.

> The question then should be why is ISC DHCP server using raw sockets?

I don’t know much about Linux network stack, but probably it’s because DHCPDISCOVER messages sent to ff:FF:FF:FF:FF:FF, and Linux network driver only accepts frames sent to the interface address? Not sure if that’s true though.

This is just a normal udp broadcast packet sent to FF:FF:FF:FF:FF:FF / 255.255.255.255. You do not need raw sockets to receive it.

There is an ISC dhcp article here with more info on why they need it: https://kb.isc.org/docs/aa-00379

But I just thought about it some more.. In a normal unix socket, you don't set the destination mac address when sending a packet. You just set the destination IP address and the kernel figures out the destination MAC address using a static, cached or dynamic ARP lookup.

But since the dhcp client has not been assigned the IP yet, ARP would fail.

The DHCP server would need to set a static ARP entry before sending the response, and hope the kernel properly fills the destination MAC. This causes more problems..

Its much easier and reliable to just craft the exact headers required as-per the RFC, especially in a cross-platform piece of software like the ISC dhcp server.

It's the fact that TCP/UDP socket interface doesn't expose macaddress, dnsmasq IIRC does the same

It's well known that iptables operates on the normal sockets layer. It would similarly be surprising behavior if you could not see packets that are present but dropped when running tcpdump. Note that this distinction only applies to packets handled locally, not forwarded.

Applications such as tcpdump and dhcp require special privileges to open raw sockets. Note that ebtables (and now by extension nftables) can be used to operate at this level.

> That is a surprising revalation; that iptables filters traffic depending on Linux implementations details. One could imagine the outcry if firewall vendor X suffered a similar “feature”. Or is this well known for Linux iptable users?

All of these major Linux firewall features have been around for over two decades and the use of multi-stage rule routing with filters is day-to-day for anyone in network ops. Cisco switches I messed with in the 90s had DHCP helpers for relaying packets across VLANs among other similar features. FTP helpers were extremely common before SSL/TLS/sftp became standard due to NAT and how the port directions work in that protocol.

iptables (well, netfilter at least) is part of Linux, so obviously it depends on Linux implementation details. Or rather its all Linux implementation details and nothing else.

Even within iptables / nftables / nft there's a ton of places to hook in. I always need to take a long hard look at this diagram to get it right: https://en.wikipedia.org/wiki/Iptables#/media/File:Netfilter...

if you have systemd/Netplan/docker, than expect chaotic firewall states.

Each can drop some nasty use-case specific assumptions that cause odd issues in other areas.

Happy computing =)

that has nothing to do with any single of those tools, just the fact of trying to manage firewall by more than one tool at once.

Netplan technically could solve it but I have zero trust in Ubuntu not fucking it up or abandoning it in few years. And starting with YAML is already begging to fail

All mentioned items have side-channel borked firewall and route rules in the past. Some bugs intermittently silently block local daemon instances from (re)loading like magic (some bugs only happen when the system is brought up).

If your daily tasks include something less borked, than consider yourself very lucky you live without systemd. If I recall, ufw was intended for simple workstation rule sets.

Personally, for home stuff I tend to use a heavily customized rule-set that interoperates with fail2ban. And a very old repeatably stable approach to setting up the interfaces from a known default state...

https://shorewall.org/

Best of luck, =)

I remember hating shorewall and similar ones because, well, I know iptables, and I know exactly what I want so using anything that tries to abstract it into it's own approach is torture as I need to take the rules I want and translate it to whatever mediocre paradigm shorewall (or ufw, or near-any other firewall manager in the wild) decided to put on top of iptables.

I ended up using ferm http://ferm.foo-projects.org/ which is basically a convenience layer over iptables, the keywords are named the same and the rules map nearly 1:1 and the changes of mapping are essentially macro and variable expansion. So it's basically iptables but a lot of tedium removed.

Our biggest one is around 1.5k rules and very manageable, using ferm with rule files generated via Puppet. Every entry gets a comment allowing us to track where it came from too.

> If your daily tasks include something less borked, than consider yourself very lucky you live without systemd. If I recall, ufw was intended for simple workstation rule sets.

Systemd has little to do with any of that

What advantages have ferm over nftables? The format looks similar.

For people who wish this weren't the case, would you want tools like tcpdump and Wireshark to also only see packets that the firewall allows? If not, then what change would you propose exactly that changes the former without breaking the latter?

Distinct socket types for monitoring vs receiving/sending.

But could it be blocked using eBPF

https://docs.kernel.org/networking/filter.html

I've just disabled those exceptional rules when dhcp isn't required.

I'm wondering if this is:

- a default configuration situation (which I think I rectified)

- DHCP exception hardcoded in the firewall binaries

Intriguing, so there's no way to block DHCP from Linux at all as all firewalls such as ufw/nftables/iptables, would use netfilter behind the scenes?

You could probably via ebtables as they inspect ethernet frames directly. That can be used for example to not allow VMs on host to spoof mac addresses.

But easiest way is just don't allow app to run with permission to access raw socket. That's it.

The problem is really that there is no way to get UDP interface that also gives you mac address of the packet so raw sockets are only way to do it.

Similarly there is no interface to send ICMP packets other than raw sockets.

Not through iptables, and probably not through nftables (though I can't find much documentation on nftables).

eBPF should still work, though. You can also configure iptables to filter based on a bpf program, combining the two. Here's an example: https://github.com/Asphaltt/iptables-bpf

Thanks a lot, will have a look at your link.

ebtables can, it inspects ethernet frames.

DHCP relies on Ethernet broadcasts to function - meaning DHCP messages are received by every NIC on the subnet. So ... already not private.

PF_PACKET is needed to look at those broadcasts because the system might not have an IP, so it can't use TCP or UDP sockets.

PF_PACKET on Linux evidently ignores iptables.

Good news: PF_PACKET requires root to use (more precisely, CAP_NET_RAW capability).

So root processes can totally ignore your firewall. This doesn't matter because:

- a firewall is really for managing external communications. If you have stuff running on localhost sending or receiving unwanted stuff, and you don't trust it, why is it running on your machine then?

- root can already simply disable the firewall by removing iptables rules or adding ones.

You can always move to IPv6 which uses multi-cast and self-generated link-local addresses, meaning PF_PACKET isn't necessary.

> are received by every NIC on the subnet

On the brodcast segment, usually L2 VLAN.

/nitpick

The netdev and bridge tables' hooks run early enough I think?

Right: https://wiki.nftables.org/wiki-nftables/index.php/Nftables_f...

"The netdev family is different from the others in that it is used to create base chains attached to a single network interface. Such base chains see all network traffic on the specified interface, with no assumptions about L2 or L3 protocols. Therefore you can filter ARP traffic from here. There is no legacy x_tables equivalent to the netdev family."

What does it even mean to block DHCP? A firewall sits between networks and filters what gets passed between them. Linux can absolutely block DHCP packets being passed between networks. But it has to see them itself, obviously. So you don't want other processes being able to see them too? Well don't run those processes. Why on earth would you run a DHCP server that you hope can never receive any requests?!

Whether or not it's sensible is not quite the point. It's surprising and noteworthy to understand. And by understanding helps understand the internals of this aspect of linux networking.

I ran into exactly this problem a few months ago. But it was a lab scenario rather than a real production one.

In my lab at work I had only a single VLAN (and couldn't get a new one added fast enough for the tight 2 day timeline of the case I was working on, though I have since done so). So I had "MAAS" (Metal As a Service) which PXE boot installs the servers plus OpenStack Virtual Machines connected to the same VLAN. Both run a DHCP service.

MAAS (using ISC DHCP Server currently) responds to any DHCP requests while OpenStack only responds to IP/MAC combinations it knows from it's configuration database and ignores others. So OpenStack wasn't impacting MAAS but sometimes MAAS was responding faster than OpenStack giving the wrong IP to VMs. As a temporary workaround I (tried) to add a firewall rule to the MAAS server to block DHCP from the specific MAC addresses of the test VMs but it didn't work.

Investigating that is when I learnt about this surprise behaviour, which was particularly surprising as I am far more knowledgable than average of many aspects of linux networking including some (but by no means extensive) kernel/netfilter internals and can write most iptables rules in my sleep from memory... despite all of that, this one was new to me :)

(As a side note, when using ovn/ovs networking, Openstack will filter external DHCP requests to VMs, which also solves this, but in this case I was specifically testing SR-IOV where the VMs are directly attached to a virtual function on the physical network card, where such filtering is not configured and may or may not be possible depending on the card)

This DHCP behaviour also happens on the client side

ARP? Don’t you need arptables for this? OSI model 101

Had an interesting exchange with chatgpt/gpt4 about this. Initially gets it wrong, but then explains it really well after being corrected with this link. Apparently Windows has similar default behavior too.

Me: Is DHCP affected by iptables firewall rules on Linux?

GPT: Yes, DHCP (Dynamic Host Configuration Protocol) traffic can be affected by iptables firewall rules on a Linux system.

iptables can be used to filter network traffic, and it does this by checking each network packet against a set of rules. If a rule matches the packet, a specific action is taken (such as allowing the packet through, dropping it, or logging it).

For DHCP, which uses UDP, the communication usually happens over port 67 for the DHCP server and port 68 for the DHCP client. If there are iptables rules that block these ports, it could prevent the DHCP from functioning correctly.

For instance, if you want to allow DHCP traffic, you might need to include rules such as these in your iptables configuration:

``` iptables -A INPUT -p udp --dport 67:68 --sport 67:68 -j ACCEPT iptables -A OUTPUT -p udp --dport 67:68 --sport 67:68 -j ACCEPT ```

The exact rules you need will depend on your specific setup and requirements. Always remember to test your configuration thoroughly to ensure it's working as expected.

Me: This suggests that answer is incorrect. https://unix.stackexchange.com/questions/447440/ufw-iptables...

GPT: I apologize for the confusion. The interaction between DHCP and iptables is indeed more nuanced than my previous message indicated.

When a machine requests an IP address using DHCP, it doesn't have an IP yet, so the DHCP server sends the reply to the broadcast IP, which is typically 255.255.255.255. Linux has a kernel parameter called `rp_filter` (Reverse Path Filtering), which by default allows these broadcast packets through, bypassing any iptables rules. This is because the kernel recognizes that these packets are part of a legitimate IP address assignment process, so it allows them in despite any firewall rules.

So in practice, while DHCP does technically use ports that could be blocked by iptables, the Linux kernel generally ensures that DHCP traffic is allowed through anyway. This behavior is fundamental to the operation of DHCP, and trying to change it could break your system's ability to obtain an IP address.

Me: How does the Windows firewall handle DHCP filtering, or not?

GPT: The Windows Firewall handles DHCP traffic similarly to other traffic, but with some built-in allowances because of how critical DHCP is for network connectivity.

By default, Windows Firewall has predefined rules to allow DHCP traffic. DHCP uses UDP port 67 for servers and UDP port 68 for clients. The firewall rules permit these inbound and outbound connections so that a DHCP client can obtain an IP address from a DHCP server.

However, unlike typical firewall rules, the rules for DHCP are handled a bit differently because, at the time of the DHCP process, the client does not yet have an IP address. For this reason, the DHCP process cannot be subject to standard IP address-based filtering rules.

In the default configuration, Windows Firewall will not block DHCP traffic because this would disrupt network connectivity. But a user could technically create custom rules to block DHCP; this would likely cause significant network problems, and therefore it's not recommended.

As a reminder, DHCP is a fundamental protocol for IP networks, enabling devices to request and receive an IP address and thus communicate on the network. Blocking DHCP traffic would typically prevent a device from obtaining an IP address, thereby blocking network access.

https://chat.openai.com/share/6e02686c-a330-4efe-b261-1e0d09...

Reply on news.ycombinator.com