Hacker News Reader

Comment by surajrmal | Hacker News Reader

Each serves a different purpose and makes different trade-offs as a result. Flatbuffers is a serialization format, not an rpc solution. grpc is optimized for rpc, whereas FIDL is meant for ipc.

Also if you are trying to be comprehensive, don't forget about AIDL/Binder which is used by Android, and mojo, which is used by Chrome. Both of these are IPC, however they are not general enough to be used outside of their respective platforms.

View on HN

It's there really some reason not to generalize IPC to RPC "between processes"?

To be more specific, is there really a justifiable specific of either of these applications which prevent us from having a single tool which would cover both?

Sure. In IPC, you can have shared resources such as memory. In RPC, you need to assume all parties may be on separate systems. In IPC, you may be able to make guarantees about message ordering that are more challenging with RPC. For instance, packets may be dropped over a network link and packets may arrive out of order. In IPC, you can express backpressure without the need for packet dropping, and therefore can avoid overhead of things like TCP. With IPC, links are trusted, so you can avoid encrypting messages, but that is not something you can safely assume with RPC. Because of these sorts of things, the cost of sending a message is relatively low (measured in us) compared to RPC (measured in hundreds of us), and so the cost of serialization may be of greater concern as it may start to dominate cost. As a result, you may optimize your serialization format differently. Beyond that, an IPC system may allow for advanced usage of the shared kernel which is not possible across different systems each with a separate kernel.

QNX makes no distinction between IPC and RPC. In fact, a program won't be able to tell if the other end of a message port was on the same CPU, same computer, or network (well, maybe by timing results). You didn't even need to recompile code to get that either. It just works right out of the box.

When you do this, you have to live by the combined constraints of both transports. It's not necessarily wrong, but having different mechanisms which are optimized for the specific constraints of each is also valid.

I wonder how many of those constraints actually apply to the IDL and surrounding programming model as opposed to the mere (de)serialization.

Could we have a single IDL that generates two sets of serialization code one optimized for IPC and the other for network transport?

COM, .NET Remote and RMI do it.

I'll give you a more concrete experience. We have protocols designed to take advantage of shared memory to allow dma directly from the hardware to the pages backing the file you end up reading from. There are many layers/processes between my program and the driver which ultimately talks to the hardware, and being able to bypass copies for the data is useful. Now the protocol encodes this expectation of shared memory usage within it. We later built a mechanism to take existing protocols and use them across network boundaries, but were unable to utilize any protocols which relied on shared memory. It may be possible to emulate shared memory across the network boundary, but it's hard to do so performantly in all cases. So rather than modify the existing protocol to avoid shared memory, negatively affecting existing use cases, we opted to create a second protocol which was optimized for network use cases.

There are more of these sorts of examples I could enumerate if you find it worthwhile.

It should be possible to encode those constraints as part of the language. For example, the distinction between sync and async calls can be represented by having a Future type and wrapping the return type in it.

Sync and async are all about cooperatively yielding control flow. However, in many cases, you may want to yield control on IPC, or to hold onto control on RPC. Yielding control depends on the sender logic. IPC/RPC depends on receiver characteristics. They are very orthogonal concepts.

QNX is able to do this because QNX doesn't understand shared memory. Everything is a message/channel.

I'm struggling to see why you'd want a specialised explicit IDL for IPC-only, as against it being an implementation detail using same address-space optimisation if services are co-located. E.g. cf OmniORB (CORBA) [1]

While you can have tightly-coupled, highly interactive IPC that would be pathological over RPC, where DSL/IDL and protocol semantics rightly focus on more loosly-coupled service interfaces with network error recovery, I question whether it is desirable even in the IPC case. Intra-service IPC within a single runtime can be as chatty as needed without becoming an Inter-service call (and potentially RPC in other architectures), so does Fuschia answer to an actual need - is there actually a valid use case for tight-coupled highly interactive local inter-service IPC that isn't better architected assuming RPC? Is the benefit of assumed IPC between co-located client/servers an actual problem that 'colocation optimised' RPC doesn't already solve well enough?

[1] https://www.omniorb-support.com/pipermail/omniorb-list/2006-...

RPCs aren't just slower for IPC, they are also missing very useful features. Like it's not possible to send an open file descriptor across RPC, but it's very possible (and very useful) to do the same over IPC. It's how something like an "open file dialog" can work without needing the app process to have full access to all the user's files.

Well, thinking around why you'd pass around a local FD among some processes via IPC, what are the typical use cases? As a source/sink in a chain of piped processes? the connections are the pipes, not the file except at the ends. Perhaps several services cooperating to differentially process parts of a file? you'd probably architect that as a FileService that would wrap the FD and trigger events or stream to individual processing services, and here there's no fundamental difference if it runs RPC or IPC (although IPC throughput can be optimised of course, but done transparently by the ORB.) Whereas an FD-passing IPC solution is locked to local processing anyway (and only vertical scalability), and if you're already paying the price to write IDL for that, isn't the RPC/IPC style solution above both more flexible (allows hz scale), not significantly more complex or locking you into yet another IDL ?

Eh? I literally gave an example - asking the user to pick a file or open a hardware resource (audio stream, for example, or USB device). The trusted system process shows the file picker & opens the file, and then just gives the requesting app that open FD.

It's a pretty straightforward & extendable way to do permission handling, rather than needing the kernel to do all of that natively. Whereas if you go the RPC route, then you're into not just a monolithic kernel but a larger-than-ever monolithic kernel as it's now doing permission management, too.

> Like it's not possible to send an open file descriptor across RPC, but it's very possible (and very useful) to do the same over IPC.

Really? I know you can use SCM_RIGHTS on a socket, if that is what you mean. In my mind IPC means shared mmap'd memory, though

Android's Binder driver can send arbitrary FDs as can unix domain sockets via sendmsg. SCM_RIGHTS isn't limited to sending sockets, it can send any FD (open file, open socket, a memfd shared memory buffer, handle to a hardware resource, etc...)

There is also the problem of binary layout. RPC values on binary size, and IPC values on latency with zero ser/des. The two binary layouts are simply incompatible. You can do zero copy with RPC, but that does not help with ser/des. FIDL uses a full uint64 just for a discriminated union tag, which is just crazy for any protocol that is going through a cable.

if it's IPC only then it avoids the ntohl conversion between different endian machines, or alignments, another one would be shared memory map, and special handling for os objects ("handles", "events", etc.).

It's a surprisingly readable doc, check it out [0].

Compared to protobufs, it's much more concerned with being more fixed width, word aligned, efficient copy-free in-memory accessable. Proto3 spends a lot of effort compressing ints on one hand but remaining flexible and allowing fields to be optional on the other.

On top of all that, there's a good deal that's specific to it's kernel, Zircon, it's permission/capabilities model, and how these messages interact with their syscalls.

I'd imagine these kinda details are going to matter a lot for a microkernel.

I don't know it as deeply but I'd say it's much closer to cap'n proto. Not sure who uses that though!

[0]: https://fuchsia.dev/fuchsia-src/reference/fidl/language/wire...

> Also if you are trying to be comprehensive, don't forget about AIDL/Binder which is used by Android, [...]. Both of these are IPC, however they are not general enough to be used outside of their respective platforms.

To be fair, Binder originates at Palm[1] and seems to have been originally intended to be a full cross-platform replacement for COM, including DCOM (intended but unimplemented) and OLE (apparently implemented but unreleased[2]).

[1] http://www.angryredplanet.com/~hackbod/openbinder/

[2] https://www.osnews.com/story/13674/introduction-to-openbinde...

FlatBuffers is more suited to IPC than any other purpose.

It was originally designed as a game asset storage format[1], wasn’t it? Which is why the reference implementation assumes trusted input by default[2], for example, unlike Protobufs, Cap’n Proto, or SBE.

[1] https://opensource.googleblog.com/2014/06/flatbuffers-memory...

[2] https://google.github.io/flatbuffers/flatbuffers_guide_use_c...

I've primarily used it as the serialization format for choice for encoding large files written to disk. The ability to mmap it and not need to read the entire file to access specific data was very useful. My understanding is that this sort of usecase is how it's typically used.

Reply on news.ycombinator.com