Not knowing the /proc file system

I was intrigued and looked for details. Thankfully though, nowadays many virtual files in /proc are generated via `single_open_size`, which is a simpler version of `seq_file` which has no such problem (to my best understanding) because it should generate the whole file contents in one go. I couldn't pinpoint the exact kernel version though, the transition seems to have happened somewhere in the 3.1x era.

I generally agree with your comment, especially coming from OpenBSD where /proc does not exist, for reasons you mentioned, as well as a purely practical one: let's not pressure the VFS for no reason.

I think the biggest issue with /proc at the moment is that it leaks in core Linux components, and getting rid of it is becoming impossible. Some years ago I worked on adding `$ORIGIN` rpath support in OpenBSD and looked at how Linux did it. Mind you, the dynamic linker (ldlinux.so) grabs the running executable path from /proc...

One thing though, you focus a lot on syscalls, but I think the challenge in recent years has been to find new ways to have kernel <-> userspace interactions _outside_ of syscall, which are cumbersome to use, rigid in structure, and practically speaking, there are only so many entries you can store in an IDT.

The work on netlink is going in the right direction IMHO. It allows userland to communicate bidirectionally with the kernel, register on specific events, etc; while using a familiar tooling with bsd sockets that is convenient from both shell and programming languages.

you're being too negative, it's as if you don't understand "worse is better".

>If you read /proc, you must "read" each file with one unbuffered kernel read to be free of race conditions.

you make this sound like it's some arcane and tricky incantation; it's totally normal, like saying "to walk, put one foot in front of the other". It's unix file I/O 101, bufferless file operations are always atomic, it's a feature. There is absolutely no problem making atomic reads from files, it's a massive fucking feature.

The point is not that this is the best conceivable way to get this information, but that by making this information available this way by default, you can write and leverage all sorts of shell scripts while you are noodling at your keyboard. You don't very often need to turn your keyboard noodling into a 6-sigma 5-nines server, but if you know how to rip through this type of keyboard noodling, I'd trust you a lot more when it comes to uptime than all the people who show up at meetings ranting about 6-sigma and 5-nines while they cite Dave Cutler.

btw linux the /proc text file tree has already been supplanted by a binary "restatement" of the same and more data. I think this is more in keeping with the standard unix/solaris /proc approach, but it's not something I've had a need to look into much.

I believe that the /proc filesystem first appeared commercially in Solaris, and was then adopted and expanded in Linux.

I got an SDF account recently, and was surprised to find it in NetBSD. OpenBSD has great resistance to it.

...looking at the wiki, many more kernels implement /proc:

"Many Unix-like operating systems support the proc filesystem, including Solaris, IRIX, Tru64 UNIX, BSD, Linux, IBM AIX, QNX, and Plan 9 from Bell Labs."

https://en.wikipedia.org/wiki/Procfs

Yup. Some parts of the API are a huge annoyance like that.

Also there's no guarantee whatsoever that /proc/123 is going to refer to a specific process, or to remain existing while you get all the data you need. The whole API is full of race conditions.

“The Linux kernel has no standard mechanism for delivering a variable-sized result from a system call.”

In NetBSD you can create XML serialized system calls for this. :-)

> The Linux kernel has no standard mechanism for delivering a variable-sized result from a system call.

Maybe we can expand the userspace networking interfaces, and make a fully featured messaging system. Some day Linux may even reach feature parity with L4!

Thanks for the info. To be honest I was aware of the fact that the `/proc/$PID` could disappear by the time `/proc/$PID/`. But felt a bit uninspired to search for a solution. I took a bit of time today to look into it and made some edits to the original post. Specifically, I tried to improve the error handling around the `fopen` call by checking for a possible NULL pointer and also setting up a call to `access` before even attempting. I wonder if this is a valid approach, or at least an improvement over the original?

Some of the /proc APIs have been plugged into netlink and can be used from there. Nicer interface if you can embed libnl or libmnl.

> If you read /proc, you must "read" each file with one unbuffered kernel read to be free of race conditions

AFAIK, as that StackOverflow page also says, that’s not guaranteed to be possible. https://man7.org/linux/man-pages/man2/read.2.html:

“RETURN VALUE

On success, the number of bytes read is returned (zero indicates end of file), and the file position is advanced by this number.

It is not an error if this number is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now (maybe because we were close to end-of-file, or because we are reading from a pipe, or from a terminal), or because read() was interrupted by a signal”

I think you can avoid the “because read() was interrupted by a signal” part by not installing signal handlers, but even if that’s an option for you, that list isn’t exhaustive.

Your best bet is to pass the maximum supported number for count to the call (SSIZE_MAX in POSIX, 0x7ffff000 on Linux, and hope that the call returns all bytes.

Luckily, I think that will be fine most of the time, but that doesn’t mean /proc is a good idea.

TIL "real" fs calls aren't system calls..

I'm not disputing there are issues but that's a very misleading description of the implementation.

I like your style, but it's not quite right:

  $ echo /proc/[0-9]/status | wc
        1       7     105
  $ echo /proc/[1-9]*/status | wc
        1     483    8778

It also will sporadically print error messages due to all the race conditions. Here's my stab at it:

  $ cat dumbps 
  #!/bin/sh
  
  case $# in
    1)
   ;;
    *)
   echo "Usage: dumbps user" >&2
   exit 2
   ;;
  esac
  
  2>&- find /proc -mindepth 2 -maxdepth 2 -type f -name status -user "$1" | awk '{
   while ((getline li < $0) > 0) {
    if (li ~ /^Name:/) {
     split($0, fn, "/")
     print fn[3], substr(li, 6)
     break
    }
   }
  
   close($0)
  }'
  $ ./dumbps "$USER" | grep -w 'bash$'
  1616678  bash
  $ ./dumbps 0 | grep -w 'systemd$'
  1  systemd
  $ 

It's pretty fast too:

  $ ls -ld /proc/[1-9]*/status | awk '$3 == "root"' | wc
      445    4005   29367
  $ time ./dumbps >/dev/null            
  Usage: dumdps user
  
  real 0m0.001s
  user 0m0.000s
  sys 0m0.001s
  $

I think you may have convinced me to learn more awk with this. I'd have constructed something horrible with sed or so to do this. And this is just much cleaner

What specifically are you referring to?

Do you mean like using `fp` for a file pointer or `fd` for a file descriptor? That is idiomatic, and I would consider calling them `filePointer` or `fileDescriptor` to be an obvious smell that the developer doesn't know what they're doing.

Just inertia from the times where teletypes were a thing. As in, a mechanical printer that served as your console. When you have that as your interface you want to keep things terse.

Even after that you had compilers with limits like 6 character names for a symbol.

Those constraints went away, but names made to be comfortable for the users of teletypes and ancient compilers stuck around, and people made more of them because it fit the preexisting theme.

And now it's just plain inertia, where it keeps on going because that's what it looks like in the books, so students imitate it.

> Tangent: Looking at those code samples, I wonder whether coming up with the shortest possible, most cryptic variable names is somewhat of a sport amongst C developers.

I definitely make a point of using only single-letter variables in most of my C and python programs.

It is a very common usage in scientific computing. In math, all variables are single letters. Always. If a variable has more than one letter, you read it as the product of several variables, one for each constituent letter. When you are translating a formula that reads "y=Ax", you want to write something like

    y = A * x

Writing this formula as

    output = operator * input

or, god forbid, as something like

    output = linalg.dot(operator, input)

is completely ridiculous to any mathematician.

Mathematics itself used to be like that in ancient times. But after centuries of distillation, we arrived to the modern efficient notation. Some "programmers" want us to go back to the ancient ways, writing simple formulas as full-sized English sentences. But they will take single-letter variable names from our cold, dead hands!

Of course, the first appearance of each single-letter variable must be accompanied by a comment describing what it is. But after this comment, you can use that letter as many times as you want. Encoding that information in the variable name itself would be disturbingly redundant if you use the variable more than once (which will be always the case).

It's party cultural. Look at the NT kernel mode documentation. In the Windows side of the world, it's perfectly normal for functions to take eight to a dozen parameters, each with a fairly verbose name.

Example:

https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

Maybe your tangent is pointing at a wrong direction, because I think there are also a sizable portion of such programmers in many other languages. It seems that anOverlyLongAndInformationFreeIdentifier is widely despised, but once names do have enough information contents, an exactly preferred name wildly varies even in a single code base. For example induction variables in Python generators tended to be shorter than average in my experience.

To some extent, I like when people do it in other languages. It exhibits strong exposure to older programming material (where overshort vars abound), evincing great passion for programming, computer science etc.

A lot of lisp material features single letter vars, for example. (Actually, var length in prod code bases seems related to scope. So toy examples with obvious context see single letters strewn about. But digging through the classic books will rub off on the budding programmer...

Because for some reason many C programmers still insist on 80-character line length limit and 8-wide tab indentation, so something has to give to fit all on the screen

You should try Prolog. That's where single-letter variables are really, really common.

C - not so much. In C you are more likely to see acronyms. It's also interesting that for some reason, macro names are spelled out in full, but function names are abbreviated. So, typical C looks like:

   LONG_SCREAMING_MACRO_NAME_(uh, oh);

Sure...all us oldsters should be using things like ThisIsAVariable_i_like_it_very_much and spend a few hours making sure our bloated IDE highlights the variables we really like in mauve. Bonus points for using ChatGPT to generate the variable names.

Same here, you can get pretty far with just catting, grepping, awk/sed/sort/uniq'ing all the various /proc entries. And not only /proc/PID/stuff, but also each individual thread (task) state from /proc/PID/task/TID/* too.

Initially I wrote a python program for flexible querying & summarizing of what the threads of interest are doing (psn) and then wrote a C version to capture & save a sampled history of thread activity (xcapture) [1]. I ended up spending too much time optimizing the C code - as just formatting strings taken from /proc pseudofiles and printing them out took very little time compared to the kernel-dives when extracting things like WCHAN and kernel stack via the proc intereface.

That's why I've since built an eBPF prototype for sampling the OS thread activity. The old approach still works even on RHEL5 machines with 2.6.x kernels without root access too :-)

[1] https://0x.tools/#usage--example-output

That's one thing I don't like about Linux. There are too many times where one has to parse things that are not a proper serialization format, and the things that ARE proper formats are a actually a mix of multiple. Sometimes you even see binary files being used to store less than 100 bytes.

It would be cool if they just said "Everything here will be TOML" or something.

But I like to stick with higher level tools anyway and avoid touching the low level stuff on Linux, so it's fine in practice.

Yeah it should be something like

    char fname[BUFSIZE];

Also better to pass that in as a variable I don't think you can return a char array allocated on the stack like that

> This requires a GNU xargs that supports NULL termination (or something compatible); as I understand it, this cannot be done with POSIX tools.

You can do this with `tr`, right?

    #!/bin/dash
    for path in /proc/*/cmdline
    do
        p=${path#*/} p=${p#*/} p=${p%/*}
        case "$p" in *[!0-9]*) continue;; esac
        c="$(tr '\0' ' ' < "$path" | sed '$s/ $//')"
        [ -n "$c" ] && printf %6d\ %s\\n "$p" "$c"
    done

Those escape sequences are covered in the standard: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t...

(But please correct me if I'm missing something!)

This also avoids (POSIX-)undefined behavior with `echo` in case one of the processes' `argv[0]` happens to begin with `-n` or any argument contains a backslash.

Yeah, so you are probably right. The custom definition is something that grew out of my response to the following two posts:

A general review of the stdlib https://nullprogram.com/blog/2023/02/11/

A review of scanf https://sekrit.de/webdocs/c/beginners-guide-away-from-scanf....

Originally I implemented `getLine` (terrible name!) as a way to get multiple lines of input from stdin in a relatively safe way. The implementation borrows almost all of its ideas from the `sekret.de` post. Because I knew the implementation and it was close to hand on my machine, I used it for this version and just swapped out `stdin` for just any old file.

Edit: here's the implementation for reference, https://github.com/adammccartney/algorithms/blob/master/libs...

AFAICT, getline() is not available on Windows, neither with MSVC nor MinGW.

The actual solution is to use C++ instead of C :)

As written, it should be `s_hasdigit`, but it's actually wrong - it should be:

  int s_isnum(const char* s) {
    int result = (*s != '\0');
    while (*s != '\0') {
        if ((*s < '0') || ('9' < *s)) {
            result = 0;
        }
        s++;
    }
    return result;
  }

otherwise, it will choke on directories like `/proc/etc64/` or `/proc/net6/` if any such directory is added. (Plus other bits like early return, but that's not a correctness bug.)