How Raw sockets behave differently in macOS and Linux | Hacker News Reader

How Raw sockets behave differently in macOS and Linux

swagnik.netlify.app

111 pointsadityaathalye41 comments

It's fascinating watching someone write code for macOS while only referring to Linux and FreeBSD man pages.

On macOS, 'man ip' gives all the necessary info about raw sockets and IP_HDRINCL:

> Outgoing packets automatically have an IP header prepended to them (based on the destination address and the protocol number the socket is created with), unless the IP_HDRINCL option has been set.

> Unlike previous BSD releases, the program must set all the fields of the IP header

> Note that the ip_off and ip_len fields are in host byte order.

In a lot of cases, the macos man pages leave a lot to be desired, or don't come up in a google search. I know I'm personally not in the habit of remembering apropros exists, so I can imagine that the first step to manpage discovery being a google search.

And most of the time the docs are interchangable enough. Until they're not. At the raw socket layer, I'd still expect the macos docs to be consulted _eventually_.

Thr stark difference between Linux and macos is that ONLY macos can filter inbound Ether packets by its destination process ID.

Becasue BSD raw socket rules.

That's how you get Sniffer to be performing waaaaay more intuitively on macos than at all on Linux.

Do you have a link to Sniffer?

You can do the same on Linux, just not easily from the command line; ebpf however can do all this and more. Once you have built it you can just create a program that you can use from the command line. There are guides out there to show you how.[0]

[0] https://www.datadoghq.com/blog/ebpf-guide/

I've been meaning to try out eBPF. I read a tutorial on writing a simple perf application in Rust, but I kind of lack the intuition to know what it's good for.

Do you know a good overview of what can be done with eBPF?

Think of it like small, isolated programs that run in a VM that is managed by the kernel and has access to many kernel APIs and datastructures without needing to do syscalls.

I believe they can both monitor and manipulate data as it flows through the OS primatives (sockets, file descriptors, etc.).

nftables supports matching by cgroups, which arguably is a better idea than process IDs.

https://wiki.nftables.org/wiki-nftables/index.php/Matching_p...

Speaking of sockets on Mac, anyone knows if there is a documentation about how the network stack on MacOs works? (Routing, firewall, resolving, kernel extensions filtering/offloading, etc).

This might be a good place to start:

https://developer.apple.com/library/archive/documentation/Ne...

Why would anyone resort to reading FreeBSD documentation when you can read the source and manpages for macOS?

I guess I was tunnel visioned and lost clarity (and also, common sense). While searching for reasons why the code (from Graham King’s blog) had worked on Linux and not on macOS, a stack overflow answer [1] from 2015, from a person who had also been on the same journey, directed me to that page [2] — that talked about raw socket peculiarities in FreeBSD. So I dived straight into FreeBSD docs.

[1]- https://stackoverflow.com/a/32599757/3728336

[2] - https://cseweb.ucsd.edu/~braghava/notes/freebsd-sockets.txt

OP, you may find this [1] “trick” useful. It allows you to dynamically determine the correct byte order for the various IPv4 headers for the platform and thus avoid the need to statically decide on the byte ordering for each platform you intend to target.

You may also find this [2] table useful, it shows which platforms allow the combination of IPPROTO_ICMP + IP_HDRINCL so it may be used without elevated privileges.

In general, my experience of raw sockets is that they are not very “raw” at all, the OS can and does still perform a variety of modifications and additions to what you send and receive, in highly platform specific and often poorly documented ways. In particular, TCP and raw sockets should generally be avoided.

[1] https://github.com/fujiapple852/trippy/blob/master/crates/tr...

[2] https://github.com/fujiapple852/trippy/issues/101#issuecomme...

Apple has a relatively new (I think introduced in 2018) "Network.framework" Swift API [1] for working with raw sockets.

Maybe this post is dealing with the layer below - I'm out of my comfort zone with networking - but I recently built a custom protocol using Network and it's working well for me so far.

[1] https://developer.apple.com/documentation/network

ICMP is lower-level than what Network.framework provides.

CFSocket would be the Swift/Obj-C API.

https://developer.apple.com/documentation/corefoundation/cfs...

There's no NWProtocolRAW class so no: https://developer.apple.com/documentation/network/building-a...

This post links to some good documentation about sniffer related options: https://wiki.wireshark.org/CaptureSetup/Offloading#checksum-...

The sysctls are net.link.generic.system.hwcksum_tx and net.link.generic.system.hwcksum_rx on macos.

This is why i f**ing hate macOS. Developers wax poetic about 'oh its a UNIX', but NOTHING works like Linux once you dig into it. Zsh is different from bash, subtle API differences, different filesystem, esp with stuff like /dev or /proc or /etc, no systemd, some cli utils have different flags, subtle differences like this.

At least Windows is completely different and there's no expectation of compatibility, but with the mac you can start off thinking things would work, but they don't.

Its been a decade since I had a Mac, but it definitely gave me that uncanny valley feeling. So close that you start expecting certain behaviour.

Windows and WSL feels better to me at least.

Indeed, I've been Linux-only for a bit over 10 years now, and when I have to do some work on a Windows or Mac for work, Windows with WSL feels a lot better than modern macos. You'll encounter the occasional alien thing when Windows details leak through, but for the most part it just behaves like you expect. If I had to pick a mac or windows pc for daily driver, Windows would be an easy choice now (although as coming of age in the 90s when MS was evil, it's mind-blowing to hear me say this all!)

I'm genuinely interested in understanding this "uncanny valley feeling" you've experienced. Could you elaborate a bit on it?

Having cut my teeth on Solaris (both flavours), AIX, HPUX and later Linux in the 90s, get off my lawn! ;)

Linux != UNIX

> "no systemd"

I'd call that an excellent feature and more true to Linux than Linux is to itself today.

Even Linux is not a true UNIX,it is UNIX-like. You can't expect macOS to be like Linux, it should be quite the other way around.

Hah, Linux deviates probably as much from UNIX as macOS does.

Some of the things you point out here are things you cannot rely on in Linux consistently, either. You can’t rely on Systemd existing in Linux. You can’t rely on bash being the shell in Linux. Depending on the users system and distribution these tools may or may not exist.

Also the difference in some of the tools CLI flags wind up boiling down to being the BSD versions of them as opposed to the GNU versions of them. Which, again, isn’t a problem isolated to MacOS.

If you’re considering all Unixes everything you said basically still applies even if you remove MacOS from the equation. Unix isn’t just Linux.

EDIT: I'm stupid and a bad reader.

What are you talking about? I'm responding directly to a comment that called those two things out.

My point was that systemd and bash are _not_ part of a linux system, and you can't count on them being there.

You're right, and I was not reading carefully (the grayed out GGP didn't help)

No worries. Also definitely not something to call yourself stupid over in your edit :)

Tip: when a greyed out comment is difficult to read, click on its timestamp. That will open a view on just that comment and its replies and it will not be greyed out in that view.

That is a great tip, I had no idea. Thanks!

I appreciate your candor, I wish it was more common.

The only practically relevant Linux environments you can't expect these things to exist are stripped down distros used for docker images, like Alpine (where you don't expect much from the system anyways) and Android (not sure about the latter, haven't used it in production in quite a while, but afair it's as non-standard as it gets).

Not true at all.

> Zsh is different from bash

Ubuntu has had a non-Bash system shell (dash is /bin/sh) since 6.10.

> no systemd

Without getting into objected-on-principle distributions like Devuan, tons of stripped-down and embedded linux distros besides Android and Alpine don't have systemd, e.g. DD-WRT, OpenWRT, TinyCore. Heck, launchd is more similar to systemd than SysV init was.

> /dev or /proc or /etc

Other than the parts specified by POSIX, the /dev filesystem is the wild west across tons of Linux and Unix systems, and expecting further standardization in behavior from it is a fool's errand (e.g. device name selection, device ID assignment order at boot, presence of metadata-symlinked directories like by-id, overlay/loopback devices ... all of these are highly variable even across modern, systemd-using Linux distros). I do miss /proc, though I don't miss the file descriptors it costs. /etc and XDG standards for configs are nice conventions to be sure, but a significant minority of Linux software breaks with those conventions (all-/opt install locations, anyone?).

> different filesystem,

Sure, APFS isn't ext3/4. Neither are XFS, ZFS, BTRFS, and so on. ext3/4 is a tenuous standard at best, and many businesses make a point of preferring other filesystems on Linux.

...like, I think you might have just been getting lucky and using a fairly similar set of Linux distros such that MacOS was a big switch. There are plenty of valid beefs with MacOS's divergences (don't get me started on an OS that claims certified POSIX compliance while providing a plethora of low-level system APIs documented to break in the presence of fork(2), or the not-quite-superset clusterfuck that is FSEvents vs. MacOS's not-quite-BSD-complete kqueue implementation, or the Sophie's choice of bizarre xattrs behavior vs. the historical accident that is aliases), but what you listed isn't even remotely standard Linux behavior, much less Unix behavior.

Having used hp-ux, sunos, solaris, aix, plan 9, various flavors of linux and a bunch of embedded OSs, VMS, and various Windoze systems I formally welcome you to the Real World.

The Real World - it's got more stuff than you can imagine.

This is false plurality - while some bank or legacy datacenter might have these OSes in production, 99% of the time, you'll only encounter Linux in a professional server environment (or the Real World as you call it).

How does that matter, though? You're complaining that macOS is different from Linux. That's the salient point. And, duh: they're different OSes. macOS is just as different from Linux as any other OS out there is. And macOS is closer to Linux than some (like Windows).

You initially said, seemingly implying this was a "false cool thing":

> Developers wax poetic about 'oh its a UNIX'

I don't think they meant what you think they meant by that. When people said that (back in the early '00s), people were familiar with Linux, yes, but they also interacted with several other commercial UNIXes that were still in wide use at the time (like those mentioned by the GP), as well as the BSDs. Being able to treat macOS like a UNIX (or, more specifically, like a BSD) was a big step up in convenience from Mac OS Classic.

No one expected macOS to be like Linux, at least not in the ways you are talking about.

You would probably feel the same if, today, you were to use Solaris. And you won't say it's not UNIX, right?

How many non-Linux UNIX operating systems have you used? It's Linux that's the weird one.

That's a weird objection. Linux isn't UNIX, technically. All of the current and former UNIXes had various incompatibilities. /proc was never standardized. /dev has some standard names, but not many. Not sure what you mean about /etc, that's up to whatever's installed in userland. systemd is more or less Linux only; it's weird to blame other OSes for not having it. (The funny thing is that systemd has functionality in part inspired by macOS's launchd.) The ext* family of filesystems (and certainly stuff like btrfs) is a Linux thing; other UNIXes used different file systems.

CLI utils are an interesting thing: the BSDs have always(?) had their own, and don't use GNU coreutils (which is what Linux uses). There wasn't really much coordination over the decades, so they went in different directions. Most (all?) of them should be POSIX-compliant (possibly after setting an env var, at least in GNUs case), so if you restrict yourself to functionality specified in POSIX, you should be fine.

I don't really like macOS, but your specific criticism of it doesn't really hold water. If you're going to hate on macOS for this stuff, then you also have to hate on all the BSDs, which seems a little silly and unwarranted.

> with the mac you can start off thinking things would work, but they don't.

If you're expecting things to work on macOS just like on Linux, then your expectations were way off in the first place. If you want cross-platform compat in these areas, you'll have to stick with POSIX. Which sucks, especially since no one really implements POSIX correctly, but that's just how it is, and has always been.

Reply on news.ycombinator.com