Hacker News Reader

Comment by btilly | Hacker News Reader

One common anti-pattern is that "event driven" can lead to a push architecture that works on paper, but is unreliable in practice.

The idea looks deceptively simple. If you need something to happen, you just broadcast the right event, someone else acts on it and everything is fine. You code it, it works in the demo, and you deploy to production.

And then you run across network failures. Missed events. Doubled events. Events that arrived while a database was down. Events that were missed due to a minor software bug. And so on. You layer on fixes, and the system develops more complex edge cases, and more subtle bugs.

Then you rewrite from a push based system to a pull based system, and all of your complex edge cases disappear. :-)

View on HN

Hehe, this is very familiar territory.

I worked with a team that were making quite a complex multi-part network manager some years ago. Development was going to be superfast because the lead developer had come up with this new framework (red flag! red flag!) based on publishing events and then subscribers picking them up and acting on them.

The problems began when the subsystem running user interaction started to need replies to specific things so it could tell what was going on with specific user interactions. Then message timeouts where needed so that pieces could react if their response didn't manifest in time.

What this framework unintentionally did, I cynically worked out after a while, was implement UDP multicast over TCP.

What I was watching happen, each time messaging problems come up, was the reimplementation of TCP on top of that. That's about when I bailed!

... damn. I'm a fan of message-driven systems (in theory, never actually worked on them in practice), but I will admit that "UDP multicast over TCP" sounds like a very appropriate description.

I need to think about it some more :)

Well if it's any consolation I don't think it's inevitable, just something to watch out for. In this case I think the problem and solution domains were poorly mapped.

So what happens when your push based system has a network outage? A retry? Stall the system? Reboot? queue the operation? what?

I think doing "event driven" without an event log for critical system functionality is probably the anti-pattern you are describing here. with my event log, worst case scenario is i need to reprocess all the messages in my system to recover from all of the above.

That said, most of the things u mentioned can be mitigated by a decent event bus with a competing consumer.

ie. isolate your critical components (write model) into small succinct peices that use highly reliable message delivery techniques. once committed, broadcast to other queues where delivery isnt that critical.

Or you can just level up and go https://martinfowler.com/articles/lmax.html

An event based system with a reliable event log becomes something that you can do a resync on. Which makes it into a pull based system in time of need.

Alternately you can have some sort of confirmation protocol. It is easy to go too far, but the kind of confirmation/resend logic that turns UDP into TCP has very much demonstrated its value in practice.

In a network outage, fail the request and let user/caller know.

All the options you mentioned are doing unwarranted smart work without context.

It's fine to make a push system, but you're most likely to still need pull mechanics anyway. If my component get a push from you, but I can see that I must have missed some events, I need to request them. You still need a heartbeat mechanics, so you know you would get all events.

Of course, if the events become irrelevant after the fact you don't need this, but only very few systems are like that.

The first concrete thing I learnt is this - implement pull first, it works 100% of the time, but may be inefficient with regards to time. Then implement push, it works 99% of the time but is much faster. But always have both running.

I'm totally in agreement with this. Both processes should also be idempotent (you should be able to pull multiple times without side effects, and push and pull should be able to happen at the same time without side effects). When everything is working well, the 'push' does all the work, and though the 'pull' runs every few minutes/seconds/whatever it never has anything to do.

This same thing applies to time-based events: your system should not assume that the process is always running, so if something needs to happen at exactly 9:00am (and it's not okay to just skip it if missed), it should be able to run anytime later with the same outcome as if it ran at 9:00am.

When I was first getting into this, it helped me to understand that events must follow one of the semantics: at-most-once, at-least-once and exactly-once -- with the trick that exactly-once is not strictly possible [1].

    There are only two hard problems in distributed systems: 
      2. Exactly-once delivery 
      1. Guaranteed order of messages
      2. Exactly-once delivery 

    -Mathias Verraes [2]

[1] http://bravenewgeek.com/you-cannot-have-exactly-once-deliver...

[2] https://twitter.com/mathiasverraes/status/632260618599403520

> And then you run across network failures. Missed events. Doubled events. Events that arrived while a database was down. Events that were missed due to a minor software bug. And so on. You layer on fixes, and the system develops more complex edge cases, and more subtle bugs.

Sure...if all the ilities weren't accounted for in the original design.

Async or out-of-process event systems have increased points of failure. If those aren't accounted for, then yes -- problems occur.

Doesn't make an anti-pattern, though.

Sure...if all the ilities weren't accounted for in the original design.

Virtually nobody is able to account for everything in the original design. Convincing yourself that you got it right is easy. Actually getting it right is HARD. It is possible. For example zookeeper seems to have. But your odds of success are very, very low.

See https://aphyr.com/tags/jepsen for many, many examples of competent people who thought they had it right, being proven wrong. Over and over again.

You're right -- getting _everything_ right out of the gate is impossible.

> You layer on fixes, and the system develops more complex edge cases, and more subtle bugs.

"Accounting for" != implementing to address.

Failure is expected in distributed systems. If your architecture allows no room to address future failure, that's the problem.

Failure is not only expected in distributed systems, it is mathematically impossible to avoid. The best that you can do is document your failure modes and what guarantees you will provide despite that.

Unfortunately, as aphyr proves over and over again, is that the guarantees we are given for how distributed software is supposed to work don't hold. Over and over again, across virtually every piece of well-known distributed piece of software. And my experience is that in house software is at least an order of magnitude worse.

However you think your software will work, it doesn't.

What are your thoughts on event-driven architecture when you don't need to cross network boundaries.

An example is inter-process communication.

It is a useful paradigm if you get it right. It is also easy to make a mess. But that's true of anything Turing complete.

See Erlang for life lessons on how to get it right.

Sounds great. But... how do you get from "a push architecture", if this is the mental model you start with, because it's obvious and natural, to a "pull based system"?

Is there any receipe for this in general? Once you start with a model of a push architecture... don't you kind of have it as the only sane model?

How is it more obvious and natural than pull? In pull, the thought is, "I need data, so I'll ask for it". That's pretty natural. And obvious.

...How do you ask for something that may or may not yet exist? Maybe some data that you don't want to care if it's about the past (already exist), abut the present and in the process of being computed, or about the future and not existing yet, not even the request for its computation is fired - yet you just want to write the same code that says "do this with this kind of data, always, whenever, wherever", I don't care if it the data does not exist yet now. Or how do you ask "how many time X happened since last time I asked"?

You'll end up implementing some kind of pooling loop, and sooner or later voila, you've reimplemented an event loop, and now you have a badly ad-hoc implemented event driven system anyway.

The only way to handle a "naturally push based system" is to accept that this is the natural way for it to be, a find a declarative way to express it as a rules based system instead of tangles of imperative event handlers. But this is really hard! So you settle for the "push based system" or "event driven" system instead quite often...

Have you ever queried a database? That's how. You just ask. If you don't get data, do deal with it.

Yes, there is a polling loop. Is this bad? Nope. You just have to understand the system you're building. Will you get data in a mostly consistent timing? Polling works. Is is sporadic? Try event driven. One size does not fit all. Event driven should not be used for everything and it's not in any way more natural than polling.

Definitely varies by the system, but, in my small-business internal-software experience, 99% of the time polling is good enough. Something feels innately ugly about polling, but it's simple, has failure modes that are easy to understand, and gets the job done.

Easy? How do you send anything in a pure pull architecture?

In a pure push, how do you ask for data to be resent?

There's trade-offs either way. No way is "best"; it all depends on what you're doing. I communicate with equipment over TCP, that I have to poll. There's _no_ mechanism for push. You just work with what you have and do the very best you can.

Our IT architects love event driven stuff but it always ends in hellish systems where you have to add retries on the publisher side and pull on the client side.

I've used events successfully for an important fraud detection system. There are a set of known events. A fraud model could hook into the event chain at any point. It would then produce an alert event with its findings. Downstream another handler listened for it. Coupled with RabbitMQ, the system is a pleasure to expand. All the fraud models are micro services. Just a start up script and boom, hooked in.

We planned for duplication of messages. Operations are idempotent. If the message already ran, we could rerun the calculation safely. With Rabbit the only duplicate messages were the ones humans sent to rerun a model that broke.

> We planned for duplication of messages

This is important. There is no "exactly once" system; you can have at-most-once where things get occasionally lost, or at-least-once where you resend things that you can't be sure have been recieved. The latter loses less data but needs methods for detecting or otherwise ignoring duplicates.

NFS goes for "idempotent operations", SMTP goes for message-IDs that can be used for deduplication.

Sounds like a system I built, but what I didn't account for is how often the RabbitMQ cluster would get tanked by my coworkers. The system fails when messages start getting dropped on the floor due to server reboots. Still haven't come up with a great solution, but what we have is "good enough" for that project's requirements. It was a great learning experience though, an uncomfortable reminder of the difference between reliability and durability, and it definitely helped inform subsequent architectural decisions.

Separate out another RabbitMQ cluster for those other applications/people?

And why are they rebooting all the time anyway??

I've written a system like that and it was also composed of microservices. I had one which was running on my computer for two months without a restart and it did not fail once. You can build robust systems on an AMQP backed MQ if you take care.

Reply on news.ycombinator.com