Fedora 31 has decided to allow (and have) giant process IDs (PIDs) | Hacker News Reader

Fedora 31 has decided to allow (and have) giant process IDs (PIDs)

utcc.utoronto.ca

29 pointsBender36 comments

Love the comments. Lennart (likely Lennart Poettering) explains why it was done, to be anonymously snarked by someone who clearly doesn't like it or him. at all. Lovely and so helpful.

I don't know if I'm a systemd fan (yeah, I'm not), but there's nothing wrong with having more than 32K PIDs. At all. If your software breaks, it's stupid and broken and needs to be fixed.

I've never understood the level of vitriol directed at systemd and Lennart. I can understand having technical disagreements, but the hate has been astonishing. And it's still going on.

Not a problem if Fedora decides this. But it is a problem if systemd decides this. Lennart needs to learn he doesn’t get to push his preferences on everyone, which this is. No it doesn’t cause problems. Yes it’s a preference, keep your hands off of it.

Well, if this does not cause problems and it improves some things, it is probably a better default, no?

Distributions using systemd may still change the default if they wish.

No, it’s a preference. Don’t make excuses.

Note that this is exactly the reason why anything Lennart does is met with vitriol. He can’t keep his hands out of the cookie jar and when challenged, he and his supporters bring on the excuses. It is not for Lennart or his supporters to decide what’s better for me. That’s for the distributors to decide. Systemd does not get to decide preferences and neither does it get to set defaults.

What do you mean a "preference"? It's a setting in a config file. The distro can take it as is, change it, or not use systemd. It could patch systemd, fork it too, although changing the config file (or probably override it in /etc) would be much easier.

Why did Fedora leave it the way upstream did? Because it's probably a good default. Every UNIX I know of has had this 20 years ago.

> cookie jar

The INIT daemon is pretty tightly involved in PIDs. So, not really hand in the cookie jar, in this case.

Audio subsystem, imho, not part of init's job. Bugs? Bloat? not good. If you want to criticize systemd, this is NOT the thing to complain over.

So because it can be changed it’s not a problem systemd is changing the default behind the distributions back?

And because there are worse things it’s not a problem?

Systemd does not get to set the defaults to Lennarts preferences. And if it tries to, it deserves all the vitriol that results.

How is it behind anyone's back? Fedora made the change by updating to a new systemd and accepting it's defaults and behavior. They could have easily changed back or to whatever else they wanted with a one line config change.

It was clearly Fedora that chose this setting for their distro, not Lennart (of course he's always personally is solely responsible, right?). Fedora just decided that as of 31 it was fine to use systemd and also have big PIDs.

It's also not a problem because behavior related to PID management is a major part of the job for the init daemon. Like it or not, init is central to the bring-up and management of userspace and processes, so after the kernel or maybe glibc, it's one of the few other packages that should reasonably own this setting. So it's not like systemd in this case is overextending into questionably relevant areas.

But vitriol for an upstream changing it's way of doing things? vitriol for an upstream even having a preference? vitriol for any reason at all about a triviality like this? For what's pretty clearly a better preference at that?

This tells me that there is baseless irrationality and unwarranted emotions going on to a level that I find absolutely shocking.

Bottom line: If someone makes something and gives it to you for free, you are way out of line to criticize them for making it they way they wanted to. You are looking at a Fedora Gift Horse that contains a systemd Gift Horse in the mouth. And then complaining that, while arguably better, it's wrong in principle for them to have given it to you they way they did.

If you don't like the thing that Lennart and his folks made and gave to you for free with no strings attached, stop using it! Go make something better. Fix it! Fork it! Stage a systemd maintainer hostile coup and lead them to push out only your desired preferences if you want.

You don't like that Fedora used systemd and allowed them to set this setting, and then gave it to you for free. Same answer.

Don't spew anger about things you got for free that no one made you take.

I don't ask this in an argumentative way. Honestly curious...

Is there a technical reason you would oppose this?

No, apart from who decides what the issues are:

* tools like top format the columns so a very long pid doesn’t fit * like with IPv6 addresses very long pids are hard to remember, type and pronounce.

Software that can’t handle 64 bit process ids is broken and should be fixed.

The Real WTF is "to make pid collisions less likely", presumably to paper over a race condition triggered when a pid gets reallocated too quickly.

No, it's because it isn't unreasonable to have more than 32,768 processes on a sufficiently large host.

I quoted from the code comment.

If you have a 64 bit PID, and guarantee they will never be reused, you have just eliminated a whole slew of race conditions and security vulnerabilities, and can probably eliminate a lot of other code designed to avoid said things.

I would go for a 128bit PID; a 64bit PID runs danger of collision if 2^32 processes have been created and killed. A 128bit PID is unlikely to ever collide and you can write code with the assumption that a PID is uniquely identifying to a process and will not be reused or collide in any reasonable timeframe (100 years or so).

The pids are allocated sequentially rather than randomly, so birthday paradox isn't an issue here.

If going for big PIDs I'd do random assignment, means people will stop doign any kind of assumption about PIDs and stops potential side-channel leakage.

You can still do random assignment but never allocate the same ID twice. There are a bunch of methods to do that quickly and securely - for example use the result of encrypting a counter with a key only the kernel knows.

There are benefits to sequential allocation - for one thing, the PID doesn't need to be stored anywhere (which can matter on very low memory machines), and optimizations can be made when there are only a handful of PID's.

"There are a bunch of methods to do that quickly and securely - for example use the result of encrypting a counter with a key only the kernel knows."

That isn't a guarantee, technically though, is it? And if you consider it guaranteed, then why wouldn't a (P)RNG be just as trustworthy? Is there a mathematical argument why it's better?

For the inner parts of a cipher (before you get into "modes"), it is guaranteed. A cipher's transformation has to be reversible, otherwise you can't decrypt. If different inputs produce the same output, then it's not reversible.

Block cipher modes like CTR, CBC, etc, are built by combining this basic, reversible building block--a permutation--with additional state. The modes provide additional features, but they necessarily require additional state so that transformations are reversible. Stream ciphers, like RC4, have implicit state, such that to decrypt you have to iterate the function from the beginning state. Modern stream ciphers are built more like CTR-mode block ciphers.

You can build reversible transformations from non-reversible functions, such as a one-way hash function, using methods like the Feistel construction (https://en.wikipedia.org/wiki/Feistel_cipher), which was the traditional method for building block ciphers.

For my DNS library I used a simple, 16-bit Feistel "cipher" to generate non-repeating QIDs.[1] Most DNS libraries shuffle a 65536-element array (e.g. using a Knuth shuffle), or keep a 65536-bit set to filter values generated by a PRNG. I preferred the cipher approach because the only state necessary was a single integer "key" for each DNS query context, which may need to send a small number of packets to resolve a record. Plus I got to try out a new technique.

[1] Non-repeating until you've cycled through all 2^16 values. But even a cryptographically secure permutation becomes increasingly easier to guess the more values you can observe. And of course if you can observe the whole series then you can always guess with 100% accuracy successive values--it just cycles again. For my DNS library I got to generate a new key for every context, which resulted in a new permutation. But for system PIDs it's self-defeating to use a new key to produce a new permutation; the whole point was to avoid having to deal with potential collisions. That's why operating systems like OpenBSD don't bother with this technique for generating random PIDs or ephemeral port numbers.

Yeah, I don't know why, but I translated "encrypt" into "hash".

>There are benefits to sequential allocation - for one thing, the PID doesn't need to be stored anywhere (which can matter on very low memory machines), and optimizations can be made when there are only a handful of PID's.

I doN't think those matter that much anymore, if the machine has low memory it won't run a lot of processes and those optimizations won't matter much and if it runs a lot of processes it's likely to have more memory.

It immediately made me think about CRIU - Checkpoint / Restore in Userspace [1], which needs to be able to restore a process with the same PID as when the checkpoint is taken. When restoring, if a process already has this PID, restoring fails.

[1] https://criu.org/

More and more I wonder if the future of criu isn't all namespaces...

They do advise using namespaces, and one of their main use cases is indeed docker container migration.

Thanks for the info. Do you know where to find recent news and follow the progress of the criu project?

Sorry, I just noticed your comment. Upvoting you, hoping that you will notice mine.

Unfortunately I don't know anything specific, I haven't followed CRIU. I noticed they did a number of releases during the last years, since I needed to compile CRIU for work. I would take a look at their website [1], it seems to list recent events and they have a mailing list.

[1] https://criu.org/Main_Page

There is quite a few fun bugs you can have. HTOP can race when it reads the process PID and then tries to kill it. If in the meantime, that process dies and a new process gets it's PID, then HTOP will kill the wrong process.

There is procfd's nowadays that let you have a consistent handle on a specific process, but it doesn't solve the problem for anything that looks at PIDs.

It also naturally limits you to hosting only 32K processes; my browser at work consumes almost 1000 PIDs mainly via a lot of threads running at the same time. At home where I have even more tabs open, it's likely that it is getting close to 5000 threads running. Of those 32K processes, only 27k remain, which is a significant chunk.

I believe PID is a bit of a dated way to refer to processes and ultimately we should move to opaque process handles (like procfd).

Do all syscalls that take a pid have a procfd equivalent?

No.

Like the procdesc system on FreeBSD, which has some rather unfortunate semantics though. (Closing last open descriptors kills processes, by default.)

One could make a collection of bugs that are revealed by (quite normal) process ID re-use. One of particular note in this context is the bug that occurs when one is using Linux control groups in lieu of proper job objects (which Linux does not have) and trying to do what on an actual first-class job object would be a "terminate all processes in this job" system call.

* http://jdebp.uk./FGA/linux-control-groups-are-not-jobs.html

* https://github.com/systemd/systemd/issues/6803

I think the author is correct that the primary likely problem is mapping pid values into DB or textual files, with fixed fieldwidth assumptions.

What this made me realize, is that because I do this so little overtly, I hadn't even considered it a high risk. I do have pids in files: /var/run/nginx.pid style. They aren't fixed width fields.

Top and the like may mis-display on a wide pid I guess, unless its set to use some truncating field but then you get a problem of mis-attributing the PID value to the truncated value, cut/paste.. BOOM

FWIW, Linux has supported large PID's for a very long time. It just hasn't been default until now. I only mention because of the wording "decided to allow". It's been allowed for a very long time.

22 bits sounds completely reasonable to me. From the word giant I was afraid they were going to be 64 bits.

I was thinking Jumbo Frames.

"1500 bytes of payload"

Everything should have an identifier that can't collide with anything including objects of different types. I think typical GUIDs are 128-bits, so best go with that.

https://xkcd.com/865/

Reply on news.ycombinator.com