Bus1 – Kernel Message Bus

I don't have much of an opinion on how good this particular proposal is, but it is important to have a sensible IPC system for Linux. I see a lot of resistance to this, with claims that Unix sockets and POSIX are good enough. As the maintainer of Rust's ipc-channel, which wraps all this stuff, I strongly disagree. The complexity of the Linux implementation of ipc-channel [1] has been absurd compared to the Mac implementation, which uses Mach [2]. Don't let the similar lines-of-code count fool you—90% of the bugs that I've seen go by have been in the POSIX backend, necessitating things like manual fragmentation to get around size limits in the kernel, weird structure alignment rules in the cmsg API, file descriptor limits, ENOBUFS stuff, etc. etc. By contrast, the Mach implementation has more or less "just worked".

At this point I don't really care what the Linux kernel decides on, as long as it decides on something. kdbus or Binder or anything would have been fine; the people who are truly hurt by kernel politics are us developers.

[1]: https://github.com/servo/ipc-channel/blob/master/src/platfor...

[2]: https://github.com/servo/ipc-channel/blob/master/src/platfor...

In the LWN article linked by another commentator:

"Tom Gundersen, one of the authors, acknowledged that having a dedicated system call to create a peer, and to perform the various other operations that currently use ioctl(), might make more sense for eventual upstream submission. Devices and ioctl() have been used so far because they make out-of-tree development easier."

So I think using ioctl is a prototyping concession.

I'm not an expert on this matter but I think it is the Android devs that want it not Red Hat it is similar to the android specific BINDER IPC mechanism but with multicast support.

I assume their proposal will include dedicated system calls and their usage of ioctl is to simplify their proof-of-concept.

My impression is that bus1 appears to have taken the (copious) feedback from the kdbus debacle and actually applied it and looked at other platforms' IPC to build a novel IPC system for Linux worth using. There's a talk [1] about the design of bus1, and comparison against IPC on other platforms where IPC is saner, and how the capability model is the right design for IPC - composable, understandable, and secure-by-default. It strikes me that the bus1 devs arrived at their design after doing the things you suggested! :)

Is there something I'm missing? What might an ideal IPC API look like to you?

[1] - https://www.youtube.com/watch?v=6zN0b6BfgLY

Other commenters have made good points, I'll mention one I haven't seen yet.

Linus has mentioned multiple times in the past that he likes to merge code it is getting used even if it's not ideal. That way, it's in the tree and can be improved instead of continuing to change outside of the kernel without the core developers' input.

That's what Android's Binder is. We also have kdbus, so we know that something like this was seen as useful before. Obviously android is used on a HELL of a lot of devices. It's not considered suitable for merge into mainline for various reasons, but much like other Android specific technology something very similar to it does make sense.

So as long as they do a decent job at this there's a good chance that it will get merged into the kernel. Overtime Google will help improve it, as well as move things out of Binder or turn Binder into a layer on top of it.

Every piece of widely used external (kernel level) code that ends up in the kernel is a win. The only other option would be to wait for the extra repository to get up to standards and developed the way the kernel maintainers would like, which probably won't happen without their constant input. but this is good for Google because it means they get more eyes on their patches, and don't have to carry as much of a load when updating versions and making new releases.

This is good for everyone.

This seems needlessly negative; and you seem to be inferring a lot about what Linus wants without him having said anything. The one thing he's asked about so far on this patch series is how it handles resource exhaustion/denial-of-service issues, and they have a reasonable reply, though there may need to be some iteration there: http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg...

The last attempt, kdbus, was definitely way too complicated and funky. It went through a few rounds of review and was never merged for a good reason.

This redesign from the ground up does a lot of what you are asking for. It derives inspiration from other, well respected, capability based systems, as well as other IPC systems for the Linux kernel, like binder, which is used on one of the most popular Linux based consumer platforms, Android.

Linux already supports the IPC mechanisms supported by most of the BSDs; pipes, Unix domain sockets, POSIX IPC. But these actually have a number of shortcomings for building reliable, efficient IPC between processes.

Another inspiration that they cited as inspiration in the announcement of their talk (https://www.linuxplumbersconf.org/2016/ocw/proposals/3819), used by the most popular consumer-oriented BSD, is Mach message ports, used on in XNU, the macOS/iOS kernel. However, those are not an ideal API to adopt wholesale, as shown by the serious vulnerabilities based on trying to integrate them with a monolithic kernel: https://googleprojectzero.blogspot.com/2016/10/taskt-conside...

I would say that they have done most of what you ask. They have investigated the existing solutions, and found them wanting. They have already gone through one design that they threw away due to it being too complicated. They have picked a model that is widely respected and implemented in one form or another on most systems, a capability based model. There's a little bit of new invention here due to the ability to multicast messages and their message ordering guarantees, and probably some room for iterating, but overall, this looks like a pretty promising IPC system compared to the fairly overwrought kdbus.

And yeah, an ioctl interface may not be the prettiest, but they've said that they're using that over syscalls just because it's easier to implement and iterate out of tree that way before getting it merged, but are willing to switch to a sycall based interface if that's preferred: http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg...

Besides the ioctl vs. syscall issue, which is pretty much just an implementation detail that can be solved with a wrapper API or done away with before merging, what do you find not in good taste about this proposal?

That's a very good suggestion. But the reality is that will never, ever happen. Linus and the Linux community as a whole exist in an echo chamber.

They never resort to looking outside their little bubble world to see how others solved problems they are trying to fix. They simply are not capable of that. They constantly reinvent the wheel and reinvent it poorly. They should have simply adopted kqueue, ZFS, dtrace, jail(), etc. but didn't. They choose to refuse to see how others solved problems and use the good solutions of others.

It's one of the most damaging things about that community.

In term of linux design ioctl are natural candidates as an underlying mechanism for multi-process communication.

Quoting Steven Doyle

ioctl can be guaranteed by the kernel to be atomic. If a driver grabs a spinlock during the ioctl processing then the driver will enter atomic context and will remain in atomic context until it releases the spinlock. http://lwn.net/Articles/274695/

cooperative multitasking in critical functioning requires thread safety and atomicity.

What, bothers me is what the fuck is an iovec! That is the most important part of the norm and yet it is not defined. My fear is that to accommodate «industry grade level of developers» they will use dynamically allocated structures vs fixed size structure. And we all know that malloc in user space is already the door to hell, but in kernel space, it is a direct nightmare.

What cringes me too is it is a distributed system (and I played quite a lot with them) and they say they have tackled the problem of global ordering of the messages. Well, be it on the network, be it on silicium, I never saw anyone achieve this feature.

I fear they are over promising and they will under-deliver

If only Red Hat actually tried to accommodate others' use cases, Linux systems would be so much better.

>On why Linus doesn't want to integrate it:

Where have you read that Linus doesn't want to integrate it ?

From the LKML response it sounded as if he wanted more explanation of how it solves certain things.

Someone on twitter suggested that Apple libxpc should be used. I never heard of it but if someone did, I'll be happy to know more about it.

... that's pretty much the point of ioctl - a syscall multiplexer in the context of a fd. 9 is pretty low - check out /dev/cdrom or /dev/dri/* if you want to see a lot.

ioctl is pretty much a generic syscall interface (other OSes have similar things). An awful lot of Linux subsystems export an awful lot of syscalls through it; probably easily a four or five figure number (literature typically claims that Linux has only a couple hundred syscalls).

See https://github.com/bus1/documentation/wiki/Message-ordering for details on message ordering.

With their "synchronize-local-clocks" approach, it doesn't.

They are using Lamport's algorithm to synchronize the clocks. However, Lamport's approach creates a _partial_ ordering, and to make that a _total_ ordering you need some mechanism to break "ties". For instance, the PID, or whatever.

The catch is that the relationship derived from this arbitrary tie-breaking mechanism has nothing to do with causality, and therefore the total order it imposes is only an artifact of the mechanism chosen.

Finding a tie-breaking mechanism that corresponds to the sending events is, in Lamport's own words, "not trivial".