Hacker News Reader

Comment by Animats | Hacker News Reader

The Linux /proc "file system" is kernel to user space communication hammered into the wrong form because "everything is a file". /proc is a system call with a fake file system API, and this matters.

The sample code won't work reliably, because it assumes that the "files" won't change while being read. If you read /proc, you must "read" each file with one unbuffered kernel read to be free of race conditions. See [1].

The Linux kernel has no standard mechanism for delivering a variable-sized result from a system call. I/O comes closest to that, so this was hammered into the file system API. The dents show. /proc does not have standard file semantics.

[1] https://stackoverflow.com/questions/5713451/is-it-safe-to-pa...

View on HN

I was intrigued and looked for details. Thankfully though, nowadays many virtual files in /proc are generated via `single_open_size`, which is a simpler version of `seq_file` which has no such problem (to my best understanding) because it should generate the whole file contents in one go. I couldn't pinpoint the exact kernel version though, the transition seems to have happened somewhere in the 3.1x era.

I generally agree with your comment, especially coming from OpenBSD where /proc does not exist, for reasons you mentioned, as well as a purely practical one: let's not pressure the VFS for no reason.

I think the biggest issue with /proc at the moment is that it leaks in core Linux components, and getting rid of it is becoming impossible. Some years ago I worked on adding `$ORIGIN` rpath support in OpenBSD and looked at how Linux did it. Mind you, the dynamic linker (ldlinux.so) grabs the running executable path from /proc...

One thing though, you focus a lot on syscalls, but I think the challenge in recent years has been to find new ways to have kernel <-> userspace interactions _outside_ of syscall, which are cumbersome to use, rigid in structure, and practically speaking, there are only so many entries you can store in an IDT.

The work on netlink is going in the right direction IMHO. It allows userland to communicate bidirectionally with the kernel, register on specific events, etc; while using a familiar tooling with bsd sockets that is convenient from both shell and programming languages.

> and practically speaking, there are only so many entries you can store in an IDT.

The number of syscalls isn't limited by this, they all go through the same interrupt and are dispatched to a specific handler function based on the value of a register.

For example this is how it works on arm64: https://elixir.bootlin.com/linux/v6.5/source/arch/arm64/kern...

And on x86: https://elixir.bootlin.com/linux/v6.5/source/arch/x86/entry/...

> One thing though, you focus a lot on syscalls, but I think the challenge in recent years has been to find new ways to have kernel <-> userspace interactions _outside_ of syscall, which are cumbersome to use, rigid in structure, and practically speaking, there are only so many entries you can store in an IDT.

There's always ioctls. You could have something a lot like /proc and /sys, with a directory hierarchy modelling various interesting things, but rather than the files having textual contents, you interact with them via ioctls. ioctls are identified by a device-specific integer, so you have masses of namespace (about 2*32 operations per device type, and you can have a lot of device types). They take or receive a block of memory in one go, so there is no parsing or formatting or worrying about inconsistent reads. I think this is even simpler than protocol-esque interfaces like netlink.

Agree, ioctls are a good replacement of /proc for exposing simple, structured information to/from the kernel.

It is still not perfect for all use cases though, especially because it forces you to operate in a _polling_ fashion.

Programs like top/iotop/htop are a good example (also, who is not guilty of abusing one liners a-la `watch -n1 cat /proc/meminfo`?). Instead of polling /proc or ioctls, you would really like to register on events, and get notifications pushed.

Netlink is well suited for these kinds of scenarios, though mostly to get notified of network interface changes last I checked.

That's an interesting point. I wonder if some sort of polling interface to ioctls would be feasible. On top of kqueue or io_uring or something maybe.

Netlink is pretty much as good as it gets here. It is explicitly designed as improvement over ioctl mess

> Netlink is often described as an ioctl() replacement. It aims to replace fixed-format C structures as supplied to ioctl() with a format which allows an easy way to add or extended the arguments.

> There's always ioctls.

Not good. Not so long ago I ran into a problem with mdadm which instead of reading sysfs used ioctls which simply didn't carry enough information for it to function properly. So, it still has bugs because of that, but it being written in C, reading from a file and parsing stuff from a string seems to cause a lot of melancholy in its maintainers, so the bug has been on slow burner for years now.

Today there is another option for doing the kernel-userspace communication: using an io_uring like interface. Yes, it's a syscall, but it's one with enough extensibility and performance characteristics. It's extensible because it's close to an IPC with the kernel (and it could be changed to be a new IPC mechanism for process-to-process communication if it isn't already). And it's performant because it's based in shared memory and is assyncronous by default.

What about stuff like udev? Or anything that has a kernel module and a controlling program running in userspace, eg. iscisiadm, mdadm etc? This way any given utility is free to implement and experiment with the communication protocol however they want.

But, more generally, if we want general-purpose communication, why not press further and have relational database-like interface, with transactional semantics, triggers, fine-grained ownership?..

> Some years ago I worked on adding `$ORIGIN` rpath support in OpenBSD and looked at how Linux did it. Mind you, the dynamic linker (ldlinux.so) grabs the running executable path from /proc...

`getauxval(AT_EXECFN)` is "better" but there are weird edge cases with `fexecve` where you only get `AT_EXECFD`, even ignoring the inevitable TOCTOU.

I am a bit blurry on the details because I worked on that almost 10 years ago to support a program that used (and abused) `$ORIGIN`.

I ended up pushing the executable filename through the aux vector to the dynamic linker, which is simar to what you mention AFAIU.

Never managed to get that merged upstream though. Glad to know there's now a _standard_ way.

you're being too negative, it's as if you don't understand "worse is better".

>If you read /proc, you must "read" each file with one unbuffered kernel read to be free of race conditions.

you make this sound like it's some arcane and tricky incantation; it's totally normal, like saying "to walk, put one foot in front of the other". It's unix file I/O 101, bufferless file operations are always atomic, it's a feature. There is absolutely no problem making atomic reads from files, it's a massive fucking feature.

The point is not that this is the best conceivable way to get this information, but that by making this information available this way by default, you can write and leverage all sorts of shell scripts while you are noodling at your keyboard. You don't very often need to turn your keyboard noodling into a 6-sigma 5-nines server, but if you know how to rip through this type of keyboard noodling, I'd trust you a lot more when it comes to uptime than all the people who show up at meetings ranting about 6-sigma and 5-nines while they cite Dave Cutler.

btw linux the /proc text file tree has already been supplanted by a binary "restatement" of the same and more data. I think this is more in keeping with the standard unix/solaris /proc approach, but it's not something I've had a need to look into much.

See the example in the original article. The article author does not know they have to do that, and wrote code which uses "fread".

I believe that the /proc filesystem first appeared commercially in Solaris, and was then adopted and expanded in Linux.

I got an SDF account recently, and was surprised to find it in NetBSD. OpenBSD has great resistance to it.

...looking at the wiki, many more kernels implement /proc:

"Many Unix-like operating systems support the proc filesystem, including Solaris, IRIX, Tru64 UNIX, BSD, Linux, IBM AIX, QNX, and Plan 9 from Bell Labs."

https://en.wikipedia.org/wiki/Procfs

The problem is Wikipedia being misleading again. They each support a proc filesystem. There is no single the proc filesystem that they support.

A case in point: FreeBSD's /proc is very different to Linux's /proc, and most of what one would go to /proc on Linux for is obtained via sysctl() on FreeBSD, with a lot less in the way of machine readable → human readable → machine readable busywork formatting and re-parsing involved.

Roger Faulkner's 1991 paper is good read. The /proc he describes looks alien if you come from a Linux only background.

https://www.usenix.org/sites/default/files/usenix_winter91_f...

By default, FreeBSD, just like OpenBSD, doesn’t have a /proc. I’m actually running a server without it.

/proc appeared in late versions of original Unix. That’s where BSDs inherited it from before ripping it out.

Yup. Some parts of the API are a huge annoyance like that.

Also there's no guarantee whatsoever that /proc/123 is going to refer to a specific process, or to remain existing while you get all the data you need. The whole API is full of race conditions.

>Also there's no guarantee whatsoever that /proc/123 is going to refer to a specific process

does using openat solve that or is there some kind of directory inode reuse occurring?

See also, pidfd, for similar reasons: https://lwn.net/Articles/794707/

No, that's usually the way to go, although I couldn't guarantee you that inodes could theoretically get reused; I never checked. I would assume you don't reuse an inode that's still in use...

“The Linux kernel has no standard mechanism for delivering a variable-sized result from a system call.”

In NetBSD you can create XML serialized system calls for this. :-)

Of all the BSDs, I'm only a little familiar with OpenBSD through its OpenSSH fame, so I'm a little surprised to hear that a (probably) somewhat related project would use XML in a system call. In other words, if an OpenSSH release introduced XML as a wire format I'd assume it to be an April fools's joke. [1] But I guess OpenBSD and NetBSD are less related than I thought.

Free, Net, and Open BSD are all relatively early forks of the same 386BSD project from the early 90s. Each project has different goals and code has diverged between them, though they frequently pull in changes from on another's codebases.

FreeBSD is the most general purpose of the 3 and the most popular. It is also the only BSD out of the big 3 that doesn't utilize a global kernel lock, allowing for modern symmetric multiprocessing similar to Linux.

NetBSD is aimed at being extremely portable. It's sort of like the "Can it run doom" of the OS world. Just take a look at their list of ports: https://wiki.netbsd.org/ports/

OpenBSD is aimed at being secure. Exactly how realized this goal is is somewhat controversial. But regardless of that, security is the stated highest priority of the development team.

There's also DragonflyBSD, which was forked by Matt Dillon from FreeBSD following some personal and technical disagreements. It's since diverged pretty heavily from the rest of the BSD family. Given its very low market share in this category of already niche operating systems, it seems more like a pet project of Matt Dillon's, though I'm sure it has serious users.

Before JSON and YAML, XML was all the rage in the sort of 2000-2005 era. Plenty of config files etc using it in the Linux desktop space.

Not sure if it’s related to this case or not but it was once much more popular even for sometimes human written files.

Solaris descendants like SmartOS were also heavy on xml configs as I recall

Like all NeXTSTEP ones.

> The Linux kernel has no standard mechanism for delivering a variable-sized result from a system call.

Maybe we can expand the userspace networking interfaces, and make a fully featured messaging system. Some day Linux may even reach feature parity with L4!

That is netlink.

https://man7.org/linux/man-pages/man7/rtnetlink.7.html

Not an expert, but I've seen some signs that netlink is used instead of ioctl lately. https://medium.com/thg-tech-blog/on-linux-netlink-d7af1987f8...

Thanks for the info. To be honest I was aware of the fact that the `/proc/$PID` could disappear by the time `/proc/$PID/`. But felt a bit uninspired to search for a solution. I took a bit of time today to look into it and made some edits to the original post. Specifically, I tried to improve the error handling around the `fopen` call by checking for a possible NULL pointer and also setting up a call to `access` before even attempting. I wonder if this is a valid approach, or at least an improvement over the original?

Some of the /proc APIs have been plugged into netlink and can be used from there. Nicer interface if you can embed libnl or libmnl.

> If you read /proc, you must "read" each file with one unbuffered kernel read to be free of race conditions

AFAIK, as that StackOverflow page also says, that’s not guaranteed to be possible. https://man7.org/linux/man-pages/man2/read.2.html:

“RETURN VALUE

On success, the number of bytes read is returned (zero indicates end of file), and the file position is advanced by this number.

It is not an error if this number is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now (maybe because we were close to end-of-file, or because we are reading from a pipe, or from a terminal), or because read() was interrupted by a signal”

I think you can avoid the “because read() was interrupted by a signal” part by not installing signal handlers, but even if that’s an option for you, that list isn’t exhaustive.

Your best bet is to pass the maximum supported number for count to the call (SSIZE_MAX in POSIX, 0x7ffff000 on Linux, and hope that the call returns all bytes.

Luckily, I think that will be fine most of the time, but that doesn’t mean /proc is a good idea.

Your best bet is to pass the maximum supported number for count to the call (SSIZE_MAX in POSIX, 0x7ffff000 on Linux, and hope that the call returns all bytes.

And then just hope your read buffer doesn’t overrun? Because 2GB is a lot of buffer to allocate for a read…

There are several special file types where the return value of `read` is guaranteed more than the general case.

For example, `timerfd_create`'s returned FD guarantees that `read` will always return exactly 8.

In practice signals don't seem to be a problem for (most?) proc files as the seq_file infrastructure doesn't bother to check for pending signals and terminate the read if so.

Naturally, only source reading will answer whether the proc file you're interested in is vulnerable to this or not.

> Naturally, only source reading will answer whether the proc file you're interested in is vulnerable to this or not.

At the moment. To be sure, you also have to keep track of changes to that source code to look for regressions, and have a mechanism to update all your deployed code if there is a regression there.

It could require multiple attempts at reading, and still doesn’t ensure progress, but I think /proc would be a lot more reliable if file contents included a good checksum.

TIL "real" fs calls aren't system calls..

I'm not disputing there are issues but that's a very misleading description of the implementation.

Reply on news.ycombinator.com