OpenBSD PF versus FreeBSD PF | Hacker News Reader

OpenBSD PF versus FreeBSD PF

mwl.io

198 pointszdw63 comments

And if you're using FreeBSD with a fast NIC, look into the pfilctl command and use a "head" input hook into your NIC driver's ethernet input handling routine. Doing this allows the firewall input checks to happen early, before the packet has been received into the network stack, while its still in DMA-mapped, driver-owned buffers. If the firewall decides a packet is to be dropped, the NIC can recycle the buffer with minimal overhead (skips DMA-unmap, buffer free, ethernet input into the IP stack, and the replacement buffer alloc, DMA map).

This allows for very fast dropping of DOS flood attacks. I've tested this using ipfw up to screening and dropping several 10s of millions of packets on 2014 era Xeons with minimal impact to traffic serving. I wrote a paper about this for a conf. that was cancelled due to COVID. I really need to re-submit it someplace.

This works for Mellanox, Chelsio, and NICs that use iflib (ix, ixl, igb, bnxt, and more that I'm forgetting).

I'm picturing that scene a few episodes into the soul art anime - I barely remember it but the main character was so overpowered, he was standing there absorbing a full barrage from a much lower ranked opponent flooding him with punches and kicks; the main character's health regenerated so quickly that the attack didn't have effects.

In 2014 I was working on a hardware appliance for a company that has something to do with packet capture, and I found an intel driver that implemented a ring buffer on a 1 gigabit Ethernet adapter that allowed me to capture line rate without dropping a single packet over the course of hours; prior to this the adapter was barely able to capture 90% of the packets. I remember marveling at the design that must have gone into it, and here too to your description of this performance improvement.

I miss that stuff. Thanks for sharing,

Would love to watch that talk (so please re-submit it somewhere!), but in lieu of that, is the paper available anywhere?

This is awesome to see [as a mostly-Linux person] - people have managed to impress the sentiment on me that the BSDs are well-suited for networking.

I never quite 'got it' - this helps. Are you aware of similar tweaks in Linux land, by chance?

I consider myself reasonably well-experienced as a systems administrator... but can't quite recall stumbling across anything quite like this.

https://en.wikipedia.org/wiki/Express_Data_Path

Thank you, I've heard of {e,}BPF but not so much applications - thank you again!

Makes sense why I haven't seen it, came out around the time I joined $CurrentEmployer, and we're stuck a couple decades behind.

IMO Linux is significantly better than the *BSDs at this particular use case.

Check out https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitig...

and:

https://github.com/xdp-project/xdp-tools

But don’t have to cobble together a bunch of arcane iptables commands and then combine bpf and other userland tools … when one can just use the clean syntax of PF especially for home use that’s a clear win.

I've used both extensively and I find eBPF+iptables (and sometimes nft) significantly more flexible and easier to use in the real world (not just simple examples) than PF. shrug

Do you have a sample or blogpost of how your setup looks? I’m keen to see how folks are using eBPF in the personal firewall space

> But don’t have to cobble together a bunch of arcane iptables commands

If you did manage to figure out the iptables commands you now have to change them over to nftables. :)

No, iptables is a perfectly functional nftables frontend

Not having to manage two rulesets -- one for IPv4 and one for IPv6 -- is pretty well a killer feature in my mind.

nftables is now almost 10 years old! It's time to forget the bad experiences with iptables.

I have -- I let the OpenBSD firewalls take care of it :P

Seriously though it's something I need to get familiar with, I do still have plenty of Linux boxes that face the public Internet and are currently dependent on iptables/ip6tables rulesets. The problem is I'm currently masking that pain with Ansible.

Linux certainly offers much better functionality overall but the tooling for this is a poorly documented and inconsistent nightmare.

There is definite lack of a declarative tool that glues it all.

Typical hardware switches and routers just have one (sometimes expanded by includes/macros but still) config syntax to control every part of networking stack.

So you can configure interface and set its vlans all in one place instead of creating a dozen of ethX.Y devices then crerating a bunch of brY bridges and then attaching the interfaces to them

In linux instead you'd be using iproute2 set of tools to configure interfaces and static routing, iptables for IP ACLs, ebtables for ethernet ACLs (or now nftables I guess), without any tool to apply/revert changes at once

Many tried doing that but IMO haven't seen anything good.

Many also try to "simplify" iptables and all it ends up is me being annoyed coz I know which iptables commands I need to run but I need to translate it back into "higher" level config syntax. One exception being ferm ( http://ferm.foo-projects.org/ ), because it keeps iptables-like keywords just expands on that, but it is iptables only and kinda superseded by nftables syntax anyway.

iptables/ebtables is deprecated even in RHEL. While people are free to continue not to transition to nftables complaining about problems with iptables after a decade of its replacement is a bit silly.

I would trade firewalld for pf in an instant.

> This allows for very fast dropping of DOS flood attacks.

OpenBSD has something very close to that w/ bpf(4) BIOCGFILDROP filter "drop".

https://man.openbsd.org/bpf#BIOCGFILDROP

https://man.openbsd.org/tcpdump#B

https://marc.info/?l=openbsd-tech&m=155228135415650&w=2

I think that's pretty different. It's less processing, but still copies packet off nic.

I had no idea! This is really cool. Have you by chance a blog about it?

This sounds nice for a specific case when you are not dealing with bandwidth based DOS.

For a bit of context regarding some of the "unfixed" issues in FreeBSD, some of it comes down to different security contexts. In openbsd, pf is pretty much root only, and root -> kernel escalation isn't a big deal. There's a bug, it gets fixed, but no hoopla. Easy to miss if you're not paying close attention. Maybe fixed incidentally as part of a refactor.

But FreeBSD also has jails/vnet, which makes root -> kernel escalation a lot more spicy. Eventually the bug gets found, some years later, although not really the result of negligence.

Michael W. Lucas is such a good author, and I just wanted to give a shout-out: "TLS Mastery" opened my eyes to the world of TLS / SSL certs in an understandable way that used interactive examples on Linux.

> And the PF syntax is the most approachable in all of open source Unix.

As someone not deeply into the BSDs and who has never tried pf, I would love to see this claim illustrated with a few examples.

Are there any “definitive” or authoritative beginner guides for those merely curious, but without a production system to test on?

I'd go for the handbook in that case:

https://docs.freebsd.org/en/books/handbook/firewalls/#firewa...

Get yourself “The Book of PF” and get to work.

Find your favorite topic in https://www.openbsd.org/faq/pf/index.html and compare to how you'd do it in your favorite firewall.

How do you do endpoint independent nat?

> And the PF syntax is the most approachable in all of open source Unix.

I actually find ipfw(8) [0] syntax more human-friendly.

[0] https://man.freebsd.org/cgi/man.cgi?query=ipfw&sektion=8&for...

I wish dragonflyBSD’s ipfw3 would get ported to freeBSD.

Explore in the direction you want to go:

https://chat.openai.com/share/e1e2a207-5fb4-4f12-9f3c-7850ff...

And then there's macOS pf too. Also different again!

I'm sure i read on a comment here recently that the latest macos is using a recent openbsd pf? As quite a heavy user of freebsd pf, I wonder if anyone knows more details on that?

On my mbp, the man page for `pfctl` contains the following:

>HISTORY

> The pfctl program and the packet filter mechanism first appeared in OpenBSD 3.0.

The same is in the current manpage on OpenBSD: <https://man.openbsd.org/pfctl>. The next line is more telling though:

> July 1, 2007

Although I'm running Monterey, so maybe it's updated in more recent versions of MacOS.

Mine doesn't have a date at the bottom, just a macOS version number.

edit: never mind, it does

Still 2007 here in macOS Sonoma 14.0

I'm just super surprised it's there at all considering no rules are defined by the OS, and nobody uses it anymore because all the firewall vendors moved over to system extensions.

when I use pfctl on macOS I see apple defined anchors.

I haven't looked in a while, but an update would be nice. When I was looking at it in the last couple years, many things in the networking stack were unchanged since the late 90s/early 2000s, so macos didn't have syn flood protection built in, and while the macos pf had synflood stuff, it only works if the macos host is strictly a firewall, using the syn protection for traffic where macos is an endpoint results in no connectivity.

If they pulled in a more recent pf from either OpenBSD or more current FreeBSD would be welcome. (And you know, a recent tcp stack would be nice too; although they've added in things like MPTCP that they'd need to port forward by 20+ years)

Solaris also uses pf as the native firewall, iirc.

oh, interesting; it was ipfilter for ages, and suddenly pf-only now

https://docs.oracle.com/cd/E37838_01/html/E60993/pfovw-compa...

Don't forget DragonflyBSD version of PF.

They look an old version of OpenBSD PF and updated it to be multithreaded.

I could be mistaken, but I believe both FreeBSD PF & OpenBSD PF are still single threaded.

https://man.dragonflybsd.org/?command=pf&section=4

The only advantage I can see to MT PF would be multiple NICs. And even then they have to be really beefy NICs

From what I understand, modern NICs have multiple Rx/tx queues and thus the work of sending and receiving can be assigned to different kernel threads.

The number of queues will vary according to the specific NIC chip.

I was going to say something similar.

Multiqueue NICs do great work on networking loads when you have 1 core per NIC queue; and you can eliminate or at least reduce cross-core communication for most of the work.

It's not complex firewalling, but I did some HAProxy stuff in tcp mode, and the throughput available when running without queue alignment was miniscule compared to the throughput available when properly aligned. Firewalling has the benefit that queue alignment should happen automagically, because of where it runs. If you're doing a lot of processing in userspace, it makes a lot less of a difference, but on a very lightweight application, there was no point in using more cores than nic queues, because cross-core communication was too slow.

DragonFly and FreeBSD have radically different ideas on how multithreading should work. (Understatement of the century right there!)

FWIW the threading in the network stack is one of the original major divergences between Open/FreeBSD PF. The way FreeBSD's PF works is within the context of FreeBSD's threaded network stack. FreeBSD PF is not "single threaded" in the way it used to mean.

I would really like to have the modern PF syntax/structure from OpenBSD in FreeBSD though. The unified rdr/nat/pass/block mechanism in OpenBSD is so much cleaner and nicer to work with.

DragonFly and FreeBSD have radically different ideas on how multithreading should work. (Understatement of the century right there!)

To elaborate for people who don't know the history: Disagreement about how multithreading should work are why Matt got booted from the FreeBSD project and started DragonFly.

It's not that simple. Disagreements are one thing, but the real reason was the conflict and drama that happened when things didn't go his way. The threading approach just happened to be a really good trigger for those conflicts.

Conflicting personalities makes resolving technical disagreements so much harder.

I was being euphemistic. It was a "doesn't play well with others" issue, but it came to the fore with SMP.

Playing well obviously important (and I wasn’t involved, have no other insight), but do you know if the poor showing for fbsd 5.x correspond w the disagreements?

FreeBSD 5.x was a very stressful time -- bringing full SMP to the entire kernel was a massive technical challenge, and it didn't help that a lot of the people who were expecting to work on it lost their jobs during the dot com crash.

5.x was a poor vintage because making the kernel SMP was hard, and social issues became problematic because SMP was hard, but Matt's departure was neither the result or cause of FreeBSD 5.x having issues.

In hindsight, with it being now 20 years later, which approach to SMP do you think was the better technical choice?

The FreeBSD model or DragonflyBSD approach?

Assuming "the DragonflyBSD approach" is the same as when it forked 20 years ago: FreeBSD's approach was definitely the right one, for two reasons.

First, adding locking is fundamentally easier to get right than rewriting everything into a message-passing model. It took ~5 years to get SMP right in FreeBSD (from when 4.0 was released to when 6.0 was released) but it would have taken double that to get things equally stable and performant with the more intrusive redesign Matt wanted to go with.

Second, pretty much everyone else went with the same locking approach in the end -- which has resulted in CPUs being optimized for that. Lock latencies dropped dramatically with microarchitectural changes between ~2005 and ~2015.

As a theoretical concept, I love the idea behind Dragonfly, but it simply wasn't practical in the real world.

FreeBSD pf has been multithreaded for a long time. The FreeBSD 11 pf(4) man page from 2013 specifically says so[1].

[1] https://man.freebsd.org/cgi/man.cgi?query=pf&apropos=0&sekti...

"SMP" might be a more useful search term. You're correct on the timeframe — merged into HEAD in 2012 https://lists.freebsd.org/pipermail/freebsd-pf/2012-Septembe...

I'm not an OpenBSD user but it sounds like this changed recently in OpenBSD-land as well, because the "Will multiple processors help?" FAQ entry disappeared some time between January and March 2023: https://web.archive.org/web/20230112170731/https://www.openb...

I didn’t know this but it makes sense. Either way for most of the use cases (homelabs on residential internet speeds) either FBSD or OpenBSD’s PF will perform just swimmingly.

Also don't forget NetBSD 's npf

I wanted to use npf but I couldn't figure out how to implement NAT64. The syntax looked nice though.

Juuuust different enough to be aggravating :P

Check the /etc/examples stuff for the best docs on `npf`. Most of what I found online was outdated or just plain wrong.

That felt like it was over before it really got started but I guess that was the point.

Reply on news.ycombinator.com