Hacker News Reader

Comment by sarchertech | Hacker News Reader

Ordering is something that everyone always forgets. The only way to actually guarantee they are sent and received in order is to use a lock on whatever subset of messages you care about being ordered. Then only process one of those at a time.

At my last company I built an ordered transaction outbox to handle this.

Each message has a partition key. When the worker picks up a new message to process it locks the partition key, so that it’s the only worker processing any messages with that partition key.

Different messages use different things for their partition key. For some messages where we only care about the order within a given order (it’s an order management system) we use the order_id. For other messages where we care about global ordering we use the tenant_id etc…

You could just use tenant_id for every message, but by using the most granular partition that you can, you can get a lot more parallelism.

View on HN

Well for starters, you should by-design not create an interface where individual discrete events are required to be delivered, without drops or strictly in order. Once you start down that path, you have to be able to answer what you do when the the destination is blocked (for any reason: network congestion or outage, service outage, etc.). This is the path where your internal queue (whether physical or virtual - that is, a cursor) starts requiring potential days of retention and the ability to stream from any point in that history. Been there, done that. You can implement that retention in all sorts of ways (rows in a database, messages in kafka, etc.) each of which has interesting cost dynamics and corner cases.

You are almost better off with a state compressed reporter that has the semantics of always reporting a _sensible_ series of events to the external endpoint where _dependent_ events have sensible relative order ("x was added, x was deleted") that converges on current state no matter how slow the outbound endpoint is and without regard to how many of them there are. No key locking and no buffering, O(1) storage per walker - just a walk on conveying state. This requires careful design of both the system _and_ the schema (boolean state needs to be accompanied with a counter, for example, to convey collapsed/consolidated history of the state changes in this model).

That's a lot more work, but it is possible, this kind of thing dates to the 80s or earlier.

The only in-order/no-drops case that is really common is logs (which is a poor way of conveying event/state tracking) and webhooks are a very poor choice for that kind of thing.

Also, in all cases, receivers should have to accept that they may receive the same notification more than once. Obviosuly there are ways around this, but they are expensive at scale.

I would really like to understand the second paragraph, but I don’t understand a single sentence. What exactly are you proposing? What’s a state compressed reporter?

The reporter is the system that sends updates to your external observers; observers register with the reporter and the reporter facilitates their consumption of events.

In the case of a webhook, this is the first tier (the actual event store) that individual webhooks contexts are following (on the service side). You need these to be per-outbound-destination to avoid head of line blocking.

I am proposing that people use state synchronization paradigms and not simple event replication for their notification systems. Since the remote side is not within your control (typically) the state compression all needs to be service side.

If you want to explore systems like this, you should start with the basics of how routing protocol peering actually works inside routing protocol implementations. In that context, you have dozens to hundreds of fast and slow consumers that are trying to converge with what is, basically a forever changing data set through the exchange of adds, deletes, and updates with peers. Any given downstream peer may block for extended periods of time, they may resume, and so on, and you cannot buffer significant amounts of data per peer (since the stack _must_ consume updates from its own peers).

Alternately, start looking into how databases sync with one another beyond the simple replicated transaction log sense.

Their comment makes no real sense to me either.

I can't tell from looking at their comment history whether or not they're AI, but their profile says 'I am nobody.'

It's a strange world.

In many cases none of that is possible because you aren’t designing the entire system. Either because it’s already been partially implemented, or because you’re modeling a system that does require ordering with no drops.

And whether webhooks are a good choice or not, they are often required for one reason or another.

> At my last company I built an ordered transaction outbox to handle this.

Almost all message queue/event streaming systems support FIFO (SQS, RabbitMQ, ActiveMQ, Kafka, Kinesis).

If you use one of these, it's easy to solve ordering.

If you only use one of those, for everything it’s easier. I still wouldn’t say it’s easy because it requires thought, and in my experience is commonly done incorrectly. I’ve regularly seen anything from picking the wrong partition key, to grabbing a bunch of messages from a single partition to process concurrently.

If you have an existing system where some services are fully event sourced, or some depend on complicated database transactions but still want to publish to a Kafka topic etc…, it gets more complicated.

This is just for arrival order though. The GP mentions transaction-based operations; handling this without massive hits to parallel processing is not an easy problem to solve.

? I'm not sure I understand.

You choose an FIFO key, e.g. order ID.

Any message with that key is processed serially. Ordering between keys is unspecified.

It is important to choose a key that guarantees sufficient ordering (for correctness), but not too much (for performance).

> You choose an FIFO key, e.g. order ID.

This is the part of the system that is doing most of the heavy lifting. Ex A, The struggles of Elasticsearch: https://archive.is/rm1UI

Could you not timestamp your message and then let the client do the ordering. Granted they might get messages out of order, but I never promised them their webhooks would be in order.

Timestamps from where? There is not necessarily going to be a central authority or processor who can guarantee these are in sync across distributed systems

Yes, you can push the problem to your users.

Regular timestamps can only be used reliably for ordering if all the timestamps come from a single system. Different systems have different clocks, so they'll never be precisely in sync. You can also have timestamp collisions, which makes ordering ambiguous.

This is assuming the use of a monotonic clock. the system wall clock may jump backwards in time. Itt also assumes that only one id can be generated per millisecond (or whatever clock resolution). In the case of multiple threads or processes on multiple cores, it becomes more likely that there will be a collision.

Timestamps don't make for a great ordering key.

Reply on news.ycombinator.com