Comment by agentgt | Hacker News Reader

Comment by agentgt | Hacker News Reader

Parent Full thread

I find this pretty hilarious coming from the guy who wrote Hystrix of which was extremely tightly coupled to another library called Archaius of which pulls a gazillion other dependencies (that being said I still highly respect Ben).

I had to spend a couple of days decoupling the mess [1].

The other annoying thing with Hystrix is that its basically Request/Reply pattern except its even worse because it uses inheritance (command pattern) and a massive singleton (HystrixPlugins). Furthermore although it uses a streaming library (rxjava) it sort of disingenuously provides very little actual support for streaming.

You don't need to have separate executables to make things decoupled but rather a good build system and architecture. Some of the ways are avoiding Request/Reply, singletons, and the command pattern.

We achieve fairly good decoupling by using a language agnostic custom message bus (it's over RabbitMQ which is AMQP... tons of clients) but you could use an actor framework or as another poster (@platform) mentioned Storm and Erlang process if you want to stick to one language.

[1]: https://github.com/Netflix/Hystrix/pull/1083

"Do as I say, not as I do". It looks to me as if Ben bundled Archaius several years ago and just recently made it a "soft dependency" which isn't required. Couldn't you say he's learned that a "gazillion other dependencies" are bad over the past several years and has learned enough to share his knowledge and implement it?

And then critiquing other parts of his code which have nothing to do with his talk is completely unnecessary.

> And then critiquing other parts of his code which have nothing to do with his talk is completely unnecessary.

It's completely relevant as its many reasons why Hystrix is very coupling which is exactly what was discussed to avoid in the presentation. I think Ben would agree and I'm sure he has learned from his experience.

The tone might sound disparaging but as I said I truly hold both the library and Ben in very high regards (otherwise why would I fix it). I suppose I have a weird since of humor because I thought hey.. I just fixed that library that had these problems and the creator is talking about those problems :) .... NOT Ben is a moron and should have done it right.

Do you have some good examples of why not to use the Request/Reply pattern? I've been trying to understand why some distributed dataflow systems prefer pushing rather than pulling.

Let's say you want to push data from the server to a mobile/web app in "real time". Examples I've come across lately would be a chat application and real time map updates (a la Uber).

Now your clients could either: (1) Pull the server every 0.5 second, or (2) Open a connection that stays open and let the server push messages across the connection.

Request/Reply and pulling is not bad in itself, but in some cases it has two downsides: (1) It delays everything because you can only pull so often. (2) I's bad for performance and scalability because all those pulls have to be handled, even when there's nothing to return.

Now you want to create a backend that also avoids pulling in the relevant places.

I think it is important to always remember that all push models, under the covers, are just pull models at a lower level of abstraction. This goes all the way down to the interrupt level.

So when we say pull models are bad for performance and scalability, what we really mean is that the abstractions we are putting around our pull model at this level are more costly than the one level down (ie your message bus is pulling off a tcp/ip socket but avoiding http).

Depending on what kind of performance and scalability you are talking about, you can either address that by going an abstraction down (which is nearly always a performance booster that comes with a development time cost) or you can pull more, less often, at the high level of abstraction (smart batching protocols also exist at every level of abstraction).

Yes this is a fundamental problem with queues, message passing, streams or any pseudo push system.

You must respect backpressure because if you don't queues blow up and the only way to respect backpressure is to have the consumer request (ie pull) for more data. The trick is to make this a lazy async pull (reactive streams) and not a blocking pull (blocking queues) ie I'm ready for more data send it to me whenever.

RabbitMQ deals with this with ACKs, Prefetch count and heartbeats (as well as some other complicated techniques like TTL and queue length).

Your point is valid. But I was under the impression that lower level stuff often did not pull. At point does a TCP/IP connection pull for example? Bellow epll/kqueue? Or the keyboard->machine connection? I haven't done anything that lowlevel in a while though, so I could be wrong.

Remember that epoll/kqueue are just more efficient abstractions on top of OS level event loops. Those event loops are polling from driver* level queues and interrupts.

Interrupts are just* the cpu polling on interrupt lines in hardware.

There are myriad of reasons (I can't find a comprehensive link) but I will try to list one or two. Some even have mixed feelings since Request/Reply is so ubiquitous and very easy to understand/implement that it makes up for it.

1.

Request/Reply is obviously not good if your problem is inherently push based ie Pub/Sub. This is because of performance reasons (could be argued) but also because Req/Rep does not fit naturally to pub/sub problems (you can think of pub/sub problems as live streams of data like stocks or chat systems).

2.

Request/Reply pattern requires very smart endpoints. For some reason this is extolled heavily in the microservice crowd because it avoids single points of a failure. However smart endpoints need to deal with server discovery, timeouts, circuit breaking, metrics, backpressure and much more. This logic often gets put into a library and suddenly becomes part of ALL clients which as the presentation mentions is bad (this is what Hystrix does to some extent). For example having all your clients depend on ZooKeeper to find some servers is pretty coupling (this is what Archaius does).

That being said the above can be mitigated by making sure communication doesn't absolutely rely on the endpoints having the same intelligence.

Message Buses like RMQ avoid this issue because the pipe is smart. Your clients don't need to have the above logic which makes implementing clients in many languages far easier... but at a big cost: single point of failure and an extra network hop (broker).

Like smart endpoint problems smart pipe problems can be mitigated as well (e.g. RMQ has cluster support).

We use a mix of both patterns and in both cases have not-so-dumb endpoints and not-so-dumb pipes.

What does the custom message bus offer over rabbitmq? RMQ seems pretty fully featured already.

The custom part isn't really custom. The custom part is just using RMQ consistently.. clients follow some guideines. Its basically saying we support Cap'n Proto, JSON, and Protobuff for the message body (one day we will pick one but alas...) and do pretty much zero routing (each message type goes to its own queue).

However we do do some endpoint routing independent of RMQ where if the client pushes a message to the "bus" and we detect the same client can consume that message we will sometimes avoid sending it over RMQ (ie local publish) (basically a performance enhancement to encourage bus usage for decoupling while avoiding the network hop for low latency).

By using protocols like AMQP and serialization that are language agnostic and focusing on doing very little routing we could switch to zmq, kafka, HTTP2 or whatever is in vogue if we wanted to.

With REST you have serious contract complexity: URIs, HTTP headers, POST form parameters, HTTP method, query parameters and the HTTP body. With an async message bus The message is the contract.

You could be more generous in your reading and assume he has learned from past errors (hystrix).

Reply on news.ycombinator.com