Hacker News Reader

Comment by WatchDog | Hacker News Reader

How many RPC/messaging protocols are there to come out of google? Protobuf/GRPC, Flatbuffers, FIDL, more I can't think of?

View on HN

Each serves a different purpose and makes different trade-offs as a result. Flatbuffers is a serialization format, not an rpc solution. grpc is optimized for rpc, whereas FIDL is meant for ipc.

Also if you are trying to be comprehensive, don't forget about AIDL/Binder which is used by Android, and mojo, which is used by Chrome. Both of these are IPC, however they are not general enough to be used outside of their respective platforms.

It's there really some reason not to generalize IPC to RPC "between processes"?

To be more specific, is there really a justifiable specific of either of these applications which prevent us from having a single tool which would cover both?

Sure. In IPC, you can have shared resources such as memory. In RPC, you need to assume all parties may be on separate systems. In IPC, you may be able to make guarantees about message ordering that are more challenging with RPC. For instance, packets may be dropped over a network link and packets may arrive out of order. In IPC, you can express backpressure without the need for packet dropping, and therefore can avoid overhead of things like TCP. With IPC, links are trusted, so you can avoid encrypting messages, but that is not something you can safely assume with RPC. Because of these sorts of things, the cost of sending a message is relatively low (measured in us) compared to RPC (measured in hundreds of us), and so the cost of serialization may be of greater concern as it may start to dominate cost. As a result, you may optimize your serialization format differently. Beyond that, an IPC system may allow for advanced usage of the shared kernel which is not possible across different systems each with a separate kernel.

QNX makes no distinction between IPC and RPC. In fact, a program won't be able to tell if the other end of a message port was on the same CPU, same computer, or network (well, maybe by timing results). You didn't even need to recompile code to get that either. It just works right out of the box.

When you do this, you have to live by the combined constraints of both transports. It's not necessarily wrong, but having different mechanisms which are optimized for the specific constraints of each is also valid.

I wonder how many of those constraints actually apply to the IDL and surrounding programming model as opposed to the mere (de)serialization.

Could we have a single IDL that generates two sets of serialization code one optimized for IPC and the other for network transport?

COM, .NET Remote and RMI do it.

I'll give you a more concrete experience. We have protocols designed to take advantage of shared memory to allow dma directly from the hardware to the pages backing the file you end up reading from. There are many layers/processes between my program and the driver which ultimately talks to the hardware, and being able to bypass copies for the data is useful. Now the protocol encodes this expectation of shared memory usage within it. We later built a mechanism to take existing protocols and use them across network boundaries, but were unable to utilize any protocols which relied on shared memory. It may be possible to emulate shared memory across the network boundary, but it's hard to do so performantly in all cases. So rather than modify the existing protocol to avoid shared memory, negatively affecting existing use cases, we opted to create a second protocol which was optimized for network use cases.

There are more of these sorts of examples I could enumerate if you find it worthwhile.

It should be possible to encode those constraints as part of the language. For example, the distinction between sync and async calls can be represented by having a Future type and wrapping the return type in it.

Sync and async are all about cooperatively yielding control flow. However, in many cases, you may want to yield control on IPC, or to hold onto control on RPC. Yielding control depends on the sender logic. IPC/RPC depends on receiver characteristics. They are very orthogonal concepts.

QNX is able to do this because QNX doesn't understand shared memory. Everything is a message/channel.

I'm struggling to see why you'd want a specialised explicit IDL for IPC-only, as against it being an implementation detail using same address-space optimisation if services are co-located. E.g. cf OmniORB (CORBA) [1]

While you can have tightly-coupled, highly interactive IPC that would be pathological over RPC, where DSL/IDL and protocol semantics rightly focus on more loosly-coupled service interfaces with network error recovery, I question whether it is desirable even in the IPC case. Intra-service IPC within a single runtime can be as chatty as needed without becoming an Inter-service call (and potentially RPC in other architectures), so does Fuschia answer to an actual need - is there actually a valid use case for tight-coupled highly interactive local inter-service IPC that isn't better architected assuming RPC? Is the benefit of assumed IPC between co-located client/servers an actual problem that 'colocation optimised' RPC doesn't already solve well enough?

[1] https://www.omniorb-support.com/pipermail/omniorb-list/2006-...

RPCs aren't just slower for IPC, they are also missing very useful features. Like it's not possible to send an open file descriptor across RPC, but it's very possible (and very useful) to do the same over IPC. It's how something like an "open file dialog" can work without needing the app process to have full access to all the user's files.

Well, thinking around why you'd pass around a local FD among some processes via IPC, what are the typical use cases? As a source/sink in a chain of piped processes? the connections are the pipes, not the file except at the ends. Perhaps several services cooperating to differentially process parts of a file? you'd probably architect that as a FileService that would wrap the FD and trigger events or stream to individual processing services, and here there's no fundamental difference if it runs RPC or IPC (although IPC throughput can be optimised of course, but done transparently by the ORB.) Whereas an FD-passing IPC solution is locked to local processing anyway (and only vertical scalability), and if you're already paying the price to write IDL for that, isn't the RPC/IPC style solution above both more flexible (allows hz scale), not significantly more complex or locking you into yet another IDL ?

Eh? I literally gave an example - asking the user to pick a file or open a hardware resource (audio stream, for example, or USB device). The trusted system process shows the file picker & opens the file, and then just gives the requesting app that open FD.

It's a pretty straightforward & extendable way to do permission handling, rather than needing the kernel to do all of that natively. Whereas if you go the RPC route, then you're into not just a monolithic kernel but a larger-than-ever monolithic kernel as it's now doing permission management, too.

> Like it's not possible to send an open file descriptor across RPC, but it's very possible (and very useful) to do the same over IPC.

Really? I know you can use SCM_RIGHTS on a socket, if that is what you mean. In my mind IPC means shared mmap'd memory, though

Android's Binder driver can send arbitrary FDs as can unix domain sockets via sendmsg. SCM_RIGHTS isn't limited to sending sockets, it can send any FD (open file, open socket, a memfd shared memory buffer, handle to a hardware resource, etc...)

There is also the problem of binary layout. RPC values on binary size, and IPC values on latency with zero ser/des. The two binary layouts are simply incompatible. You can do zero copy with RPC, but that does not help with ser/des. FIDL uses a full uint64 just for a discriminated union tag, which is just crazy for any protocol that is going through a cable.

if it's IPC only then it avoids the ntohl conversion between different endian machines, or alignments, another one would be shared memory map, and special handling for os objects ("handles", "events", etc.).

It's a surprisingly readable doc, check it out [0].

Compared to protobufs, it's much more concerned with being more fixed width, word aligned, efficient copy-free in-memory accessable. Proto3 spends a lot of effort compressing ints on one hand but remaining flexible and allowing fields to be optional on the other.

On top of all that, there's a good deal that's specific to it's kernel, Zircon, it's permission/capabilities model, and how these messages interact with their syscalls.

I'd imagine these kinda details are going to matter a lot for a microkernel.

I don't know it as deeply but I'd say it's much closer to cap'n proto. Not sure who uses that though!

[0]: https://fuchsia.dev/fuchsia-src/reference/fidl/language/wire...

FlatBuffers is more suited to IPC than any other purpose.

It was originally designed as a game asset storage format[1], wasn’t it? Which is why the reference implementation assumes trusted input by default[2], for example, unlike Protobufs, Cap’n Proto, or SBE.

[1] https://opensource.googleblog.com/2014/06/flatbuffers-memory...

[2] https://google.github.io/flatbuffers/flatbuffers_guide_use_c...

I've primarily used it as the serialization format for choice for encoding large files written to disk. The ability to mmap it and not need to read the entire file to access specific data was very useful. My understanding is that this sort of usecase is how it's typically used.

> Also if you are trying to be comprehensive, don't forget about AIDL/Binder which is used by Android, [...]. Both of these are IPC, however they are not general enough to be used outside of their respective platforms.

To be fair, Binder originates at Palm[1] and seems to have been originally intended to be a full cross-platform replacement for COM, including DCOM (intended but unimplemented) and OLE (apparently implemented but unreleased[2]).

[1] http://www.angryredplanet.com/~hackbod/openbinder/

[2] https://www.osnews.com/story/13674/introduction-to-openbinde...

You forgot AIDL, used for Android IPC between apps, and for its pseudo microkernel like functionality introduced in Project Treble.

A company that never encourages or rewards fixing things, but does reward making new things is bound to produce many versions of the same thing. How many chat systems does google have? And how many dozens more have they already killed?

“No man ever steps in the same river twice. For it's not the same river and he's not the same man.”

There can be a lot of value in rebuilding the same things from the ground up. There'll always be differences and novel things that people earlier hadn't seen. Of course it's costly and disruptive because you kill a lot of stuff, but especially in software building new things is relatively fast.

People will always complain about companies that kill a lot of products or software but at the end of the day they get more chances to build things from the ground up.

But it's not fast for the people who follow. That's the problem.

If you create a product then there is an external cost borne by the users who need to use it. And then if you replace that product with some other product, those users need to retrain.

It's really easy to create a new product, but it is a commitment for someone else to adopt it. And just because the cost is externalised, doesn't mean it doesn't exist.

> There can be a lot of value in rebuilding the same things from the ground up.

Maybe there is value for managers (who got promotions) and programmers (who got promotions and learned something), but for Google there is no value at all - it losts its users.

I remember everyone using gmail chat.. that they "remade" (killed) like two times.

Everyone just moved on.

Also I dont understand why investors arent unhappy with that. Google can still earn hundreds of millions on chat. Those wont be billions, but it is still money. And "hudred million here, hundred million there" start to add up.

> How many chat systems does google have?

* Google+ (rip)

* Google Allo

* Google Chat

* Google Groups (separate from Google's Usenet service)

* Google Hangouts

* Google Messenger

* Google Duo

* SMS via Google Project Fi

* Google Voice

> And how many dozens more have they already killed?

Too many: https://killedbygoogle.com/

You listed a number of services that are closed or have been merged together to be fair. I'm also not sure what the point of listing sms from a phone service is.

Listing the messaging App here is like comparing apples to oranges, gRPC and FIDL are for different purposes from technical perspective, people must have evaluated leveraging existing tech vs. creating a new one before the investment.

I've been only for 2-3 years ato google, butit felt like (Ads-team perspective) that protobuf is the building block in google. Some databases can hold as data as protobufs, and then convert these row oriented beasts into multi-million column databases with each protobuf leaf being separate column (!) such that analysts/linguists/statisticians and other scientists can lookup their data quickly (given some latency of minutes to hours).

You're missing the chat features of Google Maps, Google Pay, and Google Photos...maybe a few others I'm missing?

Don't forget the built-in chat in google docs that nobody uses! (Yes, the one other than the commenting system).

Oh my goodness, TIL. I know what I'll be doing tomorrow at work...

YouTube had a chat feature at one point recently.

You are not contributing to the discussion. your comment is irrelevant to the subject.

Protobuf and FIDL have very different design goals though. One is meant for backwards compatible over the wire communication between loosely coupled RPC systems, and one is meant to be a highly efficient IPC.

I expect, for example, that it is vastly more efficient to send a known datastructure over the wire in FIDL, but it is vastly more efficient to parse, modify/extend a repeated subfield, and then reserialize a protobuf.

You can't "fix" a system to achieve two goals that are in conflict, sometimes it is in fact better to have N opinionated systems, than one that is only okay at everything.

FIDL actually does make backwards compatibility a core concern. Being able to modify an existing type and allowing client and servers to continue to work when out of sync is achieved through a similar mechanism by which protobuf achieves it - a data type in which all members are optional.

That said, there are certainly many design decisions that FIDL does do differently because the constraints of the problems it tries to solve are different as you mention.

Skimming the docs, it looks like the heavyweight use use of "tables", much like Flatbuffers, seems to be the only mechanism for forward evolution of a protocol.

Looks like "structs" you can only remove boxed fields by setting their references to null, but not add fields.

Other types such as unions, enums, and bits also have properties that allow for evolution: https://fuchsia.dev/fuchsia-src/development/languages/fidl/g...

Are old applications going to run on new kernels? If so, backwards compatibility is a primary goal of an IPC dialog.

This difference in focus doesn't stop various teams at Google from forcing gRPC/Protobuf into use in places where an IPC system like Mojo/FIDL or even a simple wire protocol (JSON over WebSocket say) would be more appropriate. I know of at least 2 HW/embedded type projects at Google where major engineering had to be done for months because an upstream team refused to use anything but gRPC.

Mojo and FIDL aren't necessarily portable outside of their respective platforms. grpc may have unnecessary overhead when used for IPC, but it's well supported, understand, and portable. I don't claim to understand the full constraints the teams you refer to were under, but I'd like you to consider that they may have done what they thought was best.

Why would you assume I didn't consider?

You missed Mojo, the IPC system in chromium/ChromeOS.

Except there's also an older legacy system still used in spots, though it doesn't have an IDL it's more defined in C++ macros.

Reply on news.ycombinator.com