Comment by Animats | Hacker News Reader

Comment by Animats | Hacker News Reader

This is partly a limitation of the mediocre inter-process communication system in the Linux/Unix world. Calling another service is a Big Deal.

I've done a complex robotic system which had about ten processes running on QNX. Most of them were running some microservice - GPS, INS, LIDAR, mapping, logging, short-term vehicle control, etc. This worked fine. That's because QNX does interprocess communication well. MsgSend/MsgReceive is like making a subroutine call on the send side. The receive side is more like an event loop.

This sort of thing is common in robotics. ROS does something similar, although the interprocess communication is slower. Usually you have dummy services for simulation purposes, so you can run the operational code in a simulated environment. We could run the system for real, or run it entirely with simulated inputs and outputs, or could put the robot vehicle up on blocks and run the system with fake inputs while operating the real vehicle, engine running and wheels spinning but going nowhere. Everything could be run on one desktop, or on the vehicle's own computers, or partially split. There were shell files to launch the various configurations. We could plug in shims between services and watch the data go by.

I don't see the fear of multiple intercommunicating processes. Even on Linux, there are decent ways to distribute. They're not as good at hard real time as QNX, but they work.

Very little of this blog post is concerned about performance. We're talking about microservices running on different hosts, and IPC takes the form of a network socket, and typically HTTP.

We're also talking about very, very large scale systems, like hundreds of API endpoints, not ten, with multiple on-call rotations, each assigned their own section of the microservices surface, running in front of customers at all times, without the engineering effort to make sure things are close to perfect before exposing to customers.

What happens here is a combinatorial explosion. 1 faulty service can end up affecting it and its transitive dependencies, but operationally, it's hard to distinguish symptom from cause. Worse, a problem on one team may be cause by an unmonitored problem two teams across the network, increasing time to resolution, as each on-call is woken up to discover the root cause is another layer up the network.

More efficient RPC doesn't solve any of this.

> Even on Linux, there are decent ways to distribute. They're not as good at hard real time as QNX, but they work.

Out of interest: what would you consider to be some of those decent ways, and what are some of the less-than-decent ways?

It would be interesting to see a IPC system that abstracted away where the other process was running or treated processes running on an entirely different machine or VM as another local process.

Since scalability is probably a bigger concern in web architectures than robotics, expecting to be able to use IPC to communicate between two services is unlikely. I would imagine in many web architectures being able to scale systems independently and dynamically makes up for the overhead in communication protocols.

You can get this kind of decoupling from a pub sub msg bus. And async too. Much better than point to point HTTP.

ROS is pub/sub, but that works because most of the data traffic is one way, sensor->compute->actuator. ROS isn't usually used for function call type operations. If nobody's listening to a sensor, that doesn't hurt the sensor any.

> It would be interesting to see a IPC system that abstracted away where the other process was running or treated processes running on an entirely different machine or VM as another local process.

Erlang processes do this if I'm not mistaken.

Erlang does do this. In fact, it dawned on me that OTP is basically a system for writing "micro services" inside the app itself, complete w/ monitoring, queues (mailboxes) & supervision.

Developers are forced to export public module functions to provide a smooth api for other services (processes, in the same runtime, or distributed).

Thankfully, Erlang provides the protocols and so on, instead of requiring some nasty hand-crafted json weirdness over http.

This was attempted back in the late 80s/early 90s with RPC (from Sun) and CORBA (not sure who is responsible for that monstrosity).

This time around it's what ... 0MQ and Protobufs?

> It would be interesting to see a IPC system that abstracted away where the other process was running or treated processes running on an entirely different machine or VM as another local process.

Forcing everything into the same abstraction can have issues. For instance, calling a different machine has orders of magnitude more overhead and different error scenarios than making a call to a local process. If you make them look identical, you obscure that.

Try ZeroMQ.

Reply on news.ycombinator.com