Linux /proc/pid/stat parsing bugs
openwall.com
openwall.com
Plain text interfaces lead to complicated, potentially insecure code (especially in C!), they're prone to race conditions and slow.
I wish it was possible to retrieve that information using real syscalls. I think it's a better approach than, for example, inventing a faster way to read procfs: https://lwn.net/Articles/813827/
it sounds fishy, but just because sysctl is a mess doesn't necessarily imply that structured kernel interfaces are a bad idea
[edit:] In order to get jc to return an error one has to actually read the regex. Here is a file name that gets it to return an error:
bad) S 1 2 3 4 5 % echo '2001 (my (file) with) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc
{"pid":2001,"comm":"my (file) with","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"}
Edit: looks like I can tighten up the signature matching regex for the "magic" syntax per the issue found above. The greedy regex matching for the parser does seem to work fine, though. $ echo '2001 (bad) S 1 2 3 4 5) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc-pid-stat
{"pid":2001,"comm":"bad) S 1 2 3 4 5","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"}
But the "magic" signature doesn't recognize it: $ echo '2001 (bad) S 1 2 3 4 5) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc
jc: Error - Parser issue with proc:
ParseError: Proc file could not be identified.
...
I can fix the "magic" signature (regex) to account for such cases.[1] https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/...
Currently, but if this idea started when Linux was become popular the real data format would have been XML. It might have been nice at the time, but today we would have laughed at it and said how outdated and silly it looks probably.
Realistically, however, a reasonable format here would be CSV/TSV with a bit of escaping, or even something like netstrings[1]. Either would be unambiguous, while retaining the shell-tool-friendly nature of these pseudofiles.
I like it. To change the PCI IDs for a driver for example, change the IOKitPersonalities property in the <Driver>.kext/Info.plist file.
% strace uptime 2> /tmp/strace && grep proc /tmp/strace
17:35:24 up 3 days, 7:47, 1 user, load average: 2.29, 1.85, 1.56
openat(AT_FDCWD, "/usr/lib/libprocps.so.8", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/self/auxv", O_RDONLY) = 3
openat(AT_FDCWD, "/proc/sys/kernel/osrelease", O_RDONLY) = 3
openat(AT_FDCWD, "/proc/self/auxv", O_RDONLY) = 3
openat(AT_FDCWD, "/proc/uptime", O_RDONLY) = 3
openat(AT_FDCWD, "/proc/loadavg", O_RDONLY) = 4
struct uptime_t u = {
.time = 0
};
ioctl(open("/proc/uptime"), GET_UPTIME, &uptime);You can still have easy abstractions while providing a way around them for times they don't work well (acquiring structured data)
eBPF recently added the ability to look through internal data structures through iterators [0] so instead of parsing text we can run a program that traverses through all the task_structs and pushes the exact information we want to userspace in the form the developer wants.
So, alongside other tradeoffs, it's more flexible than syscalls.
[0] https://developers.facebook.com/blog/post/2022/03/31/bpf-ite...
In return, the kernel side API for sysfs is also a lot cleaner and allows to more-or-less expose individual variables as tuning knobs for a driver.
Of course there are edge cases, and there are e.g. some binary interfaces as well (e.g. for providing direct register access, or implementing a firmware upload interface for a device).
ABI compat issues aside, I think that implementing "a standardized [structured] record format" as suggested in the comments here is a rather bad idea, going into exactly the wrong direction by adding complexity rather than reducing it, which would definitely cause even more parsing related issues in the long run.
I'd rather have structured file than to have open 30k files (for say conntrack)
Hell, just example from the article, /proc/<PID>/stat has 52 parameters. That would be 52 opens and reads with single value per file.
> ABI compat issues aside, I think that implementing "a standardized [structured] record format" as suggested in the comments here is a rather bad idea, going into exactly the wrong direction by adding complexity rather than reducing it, which would definitely cause even more parsing related issues in the long run.
It's literally the opposite. You have to implement it once on kernel side and once in userspace vs every special format that currently needs
If you're on a system with huge numbers of connections, reading from /proc/net/tcp get extremely slow. Modern tools query connection state using netlink instead (ss vs netstat). This was done by necessity: /proc/net/tcp actually doesn't work at scale.
I agree with you, serializing and deserializing files with records is a terrible idea - it cannot be performant. JSON fixes parsing ambiguity but at a cost of being even slower. We already know it won't work.
We have already solved this for specific parts of /proc and it works great. All we have to do is finish the work and provide the rest of proc via netlink as well (or whatever else similar non-text based system for querying structured records)
CBOR could be another option: https://en.wikipedia.org/wiki/CBOR
You can't read /proc/net/tcp within a reasonable amount of time on a system with hundreds of thousands of connections. Even allocating/churning memory to store a textual representation becomes a problematic overhead.
I actually did think about this before posting my original response, and I think this is unrealistic from a practical perspective. To elaborate a bit on that:
First of, a one-size-fits-all structured format is a lot more complex than a directory with ASCII files in it that each store an integer and IMO invites itself to feature creep (i.e. more complexity).
Complexity is IMO the root cause of the issue originally discussed here (if not most bugs). The more code, the more complexity, the more bugs. In my experience, software will always have bugs, complex software more so.
There can never be a "one-true-implementation" for userspace. Because of the complexity, people will write their own ad-hoc versions. They "only need that one thing" and don't want to drag the whole library dependency in. Some people think they know better and write their own "lightweight/suckless/..." versions because the kernel one is "bloated", or "that API sucks". Some will rewrite it in their favorite programming language for whatever reason. NIH syndrome, bike shedding, ...
Then, what if a widely used implementation has a bug? Especially if it's the "one-true-library" itself? You now need to roll out a fix. Across countless Distros, embedded devices that might get maintenance updates every couple years at best, set-top boxes, network appliances, ... You'll have programs floating around that are statically linked against a specific version of the one-true-library. In the end, we have a variety of differently bugged parsers in use, simply because of the spread in versions alone. The original problem that we wanted to solve, remains.
Of course there are issues with the more simplistic approach, but in the case of e.g. sysfs, those are typically corner cases. Adding a one-size-fits-all special, structured format for everything introduces a whole lot of unneeded complexity everywhere else as well. A "one-true-format to solve all problems" that needs special library code for processing IMO introduces a whole lot more problems than it solves.
You can't really prevent that. People do funny nonsense in other self-describing data formats like JSON and XML all the time too. There's only so much you can do with a framework.
But /proc is... extremely old, and very heavily used by userspace. In practice it's never going to change.
Sure but you will get more of that if the convention is too simplistic. "one file per value" breaks really fast, just cat /proc/net/nf_conntrack or even just proc/<pid>/stats and see just how many values single entry (file/connection) has.
Doesn't need to be some ASN.1 monstrosity, could be simple conventions like "this is how key/value proc/sys file should look, this is how tabular file should look etc."
Make all escaping use same syntax, make every table separator be \t etc.
> But /proc is... extremely old, and very heavily used by userspace. In practice it's never going to change.
eh, just mount it in /proc2
Then you could just have "load a single value" function that does the unquoting, "load K/V" function for stuff like /proc/meminfo, and "load table" for stuff like /proc/<pid>/stat. Maybe "load records" for stuff like /proc/net/nf_conntrack which is essentially list of KV pairs.
All sensible ones allow you to just pass an array of parameters to command execution and not worry about spaces in them
run(f”command {arg} -v -p{opt} {target}”)
to run([“command”, arg, “-v”, “-p”, opt, target])It's an array of string parameters.
No semantics! Just an variety of customs about what it means when a parameters begins with a - or a -- or if you have -- by itself preceding some characters, how to break lists of arguments with separators, what happens when you pass the same argument twice, etc etc etc.
To choose the worst possible solution better than that, we could instead be passing in a single string with a JSON dictionary that says things like '{ "recursive": True, "force": True, "files": [ "file1", "file2", "file3" ]}'
The way most sudoers files are set up, if you're in the wheel or sudo group, you're only a "sudo -i" from a root command prompt, so I'm not sure I see why this is a vulnerability. Can anyone elaborate?
/proc/<pid>/maps is similarly frustrating: there's no clear distinction between "special" maps (like the stack) and a file that might just happen to be named `[stack]`. Similarly, the handling for a mapped region on a deleted file is simply to append " (deleted)"[1].
[1]: https://github.com/woodruffw/procmaps.rs/blob/79bd474104e9b3...
Time to let go of the everything is a stream of unorganized characters
$ cat /proc/2001/stat | jc --proc
{"pid":2001,"comm":"my program with\nsp","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"}
[0] https://kellyjonbrazil.github.io/jc/docs/parsers/proc_pid_st.../s
Here's that commit, it has a comment with an overview of the kernel limits and caveats involved: https://github.com/git/git/commit/2d3491b117c6dd08e431acc390...
As pointed out in https://news.ycombinator.com/item?id=34098360, and contrary to the proc(5) man page, this assumption is incorrect for kernel threads.
But you're right that a more general parser would need to ignore what proc(5) has to say about the limit, and parse up to a limit of 64.
As far as I can tell the difference is because when you call prctl(2) with "PR_SET_NAME" it will get truncated to the "TASK_COMM_LEN" that proc(5) discusses. See this code in kernel/sys.c: https://github.com/torvalds/linux/blob/493ffd6605b2d3d4dc700...
This is the linux.git commit that changed it, before that kernel worker threads had to obey the same limit, it was first released with linux v4.18: https://github.com/torvalds/linux/commit/6b59808bfe482642287...
For the purposes of that code it's still OK. It would read a kernel thread's file, fail to find the ending ")", and stop looking.
It's possible in principle that the kernel could a crafted kernel thread name that contained ")", followed by e.g. " X 12345 ". In that case I'd misinterpret that "12345" as the parent PID, and continue walking up that parent chain.
But in practice the kernel doesn't have, and is exceeding unlikely to have such "comm" fields.
Still, it's annoying that this was silently changed in v4.18 without a corresponding documentation update. It's easy to imagine C code written to assume its promises are true that would misbehave or segfault in the face of these longer kernel thread names.
Edit: I submitted linux-man patches to clarify this point: https://lore.kernel.org/linux-man/cover-0.2-000000000-202212...
https://github.com/pixie-io/pixie/blob/bd82bb48ef4da7d6b05f2...
https://gitlab.com/psmisc/psmisc/-/blob/master/src/pstree.c#...
But what if comm contains newlines?
Besides, it wouldn't work, because you don't know in advance how many fields are there.
Like the post says: read the whole thing into memory and do a reverse search for the last ')', i.e. strrchr
Once you are aware of the problem, it's obvious how to solve it, but I do agree that the hidden danger here is not immediately obvious at first.
This might just be for certain kernel things though. I don't see any regular processes that aren't truncated, but I see a bunch of kernel things that have more than 16 chars on my system.