Hacker News Reader

Comment by politician | Hacker News Reader

I don't understand the purpose of at-most-once semantics in practice: So, you've got some process where you don't care if the message goes through, but you're willing spend money on the compute/storage for it anyway? Why bother?

If you're designing a system with those semantics, is that because you're hoping that it's exactly-once: 99.999% of the time -- wink, wink, nudge, nudge so that your handlers don't have to be idempotent? What's the fallback plan for the day that all of your messages go into a networking blackhole?

View on HN

Soft real time data. Anything that would be out of date by the time you had timed out and resent. For example a packet of range data from a robot. Drop one? Worthless now, but another one will be along in 50 msec or so.

Telemetry from an external data source wouldn't be delivered from a queue, it'd be published into a queue. Think IoT into an AWS Lambda instance -- you've already paid the micro-cents to handle the request, so why not just forward it into Kinesis? (How does putting that data into a lossy queue save you time/money/effort?)

EDIT: Misread your example. The idea is that the robot has an outgoing buffer containing time-series data that it tries to publish on a best-effort basis? In that example, it seems like a ring buffer would be a better solution than a messaging system.

Another example of this sort of behavior was the netcode common to Quake and many other shooters--since lots of history is sent as full state instead of deltas, it's "okay" to have lost packets.

It's not ideal, and it'll screw up certain types of things that really, really want a contiguous history of events, but for normal play it's perfectly fine.

When doing client-side prediction, then it may become useful to have historical data, but as my sibling post suggests, a ring buffer of length one minimizes latency.

It actually might be interesting to see a writeup on multiplayer games viewed through the lens of eventually-consistent distributed databases.

A ring buffer of length one minimizes latency.

Why would you use a queue instead of a topic for a use-case like this?

From the point of view each of the subscribers to a topic, they see a queue.

> I don't understand the purpose of at-most-once semantics in practice: So, you've got some process where you don't care if the message goes through, but you're willing spend money on the compute/storage for it anyway? Why bother?

For instance because the time for retransmission is higher than what someone (say an HTTP request) is going to wait for.

I could imagine at most once being good for something like a ping or a heartbeat. Or even a "this is the current version of some piece of data". And if your consumers are prone to failing to ack and you don't want to gum up the queue with retries.

But your point stands that it is certainly not what we want in the lions share of use cases. I guess that's why the default semantics are at least once. With at most once being a configurable option.

I work in SMS sending, and every link in the chain has to reason about how they fail. If there is a failure, it is better that the message is dropped than that it gets resent duplicated.

It would be a bad system indeed if a failure to write a log line (say) caused a retry loop that resent the message again and again...

Many at-least-once systems support dead letter queues natively, and if not, support isn't hard to add on yourself.

I want a message queue to hold credit card authorization requests. I want to ensure that the authorization requests occur at most once, since having a customer's credit card charged twice is infinitely worse than not charging the card.

If the request never goes through, then I can always get the customer to try again, but if it gets charged more than once, then it's a huge headache, and our customers get extremely angry and lose confidence that we know what we're doing.

That is certainly doable but you need to separate the message semantics from the credit card semantics. So lets say your authorization includes a nonce which is a blend of the transaction id data such that it is unique to the transaction. Now you send that through the message system which delivers it one or more times, but the nonce identifies all of them as the same transaction so the credit card processor tosses the 'extra' ones away, it has already done the work. In such a system the message passing infrastructure needs to be at-least-once and the protocol involved insures that nothing bad happens if you deliver messages twice. Then you can route around damage or non-functioning nodes.

The reasoning at the protocol level then goes, "If I have never seen this nonce, then the transaction has not been authorized" (very durable, easy to implement), and "A request seen multiple times will only be recorded once" (also very durable and easy to implement).

But if the message infrastructure tries to guarantee at-most-once, and the user writes their protocol, "if I see this authorization I should do it, because the protocol will only send it to me one time." Then there will come a time in that protocols future where something blew up the invariants in the message infrastructure and the message arrives twice, and the wrong thing happens on the credit card because the protocol trusts it to do the right thing.

Back in the RPC wars people were arguing that it was so much "simpler" and more "transparent" if a remote procedure call had the exact same semantics as a local procedure call. And that is true it would be, if you could manage that similarity, except it has been proven over and over again that network messaging systems can't completely get it correct.

Interesting, thanks for the insight. We are doing something similar to the nonce that you mention, but your argument that at-least-once semantics is more appropriate is very convincing to me, as is your RPC argument. I'm old enough to have had to endure both ONC and DCE rpcs :)

For this kind of issue I stick to at-least-once, and then use a lot of band-aid:

- Each worker has to hook to a database where a record keep the state of the transaction: ready, processing, success or failure. With a good old SELECT FOR UPDATE a transaction will never be done more than once.

- Timestamps and alerts ensure that if the worker fails dramatically (leaving the task as "processing"), a human will check and update the task accordingly.

It has proven pretty resilient so far... but at very small scales.

If you've got a worker node that proxies those requests to your payment processor and it dies, then you can be in an unrecoverable state where you won't know whether the authorization request has succeeded or whether the message was lost.

An at-most-once policy doesn't save you from that, but an at-least-once policy would force you to write an idempotent handler that may be able to recover from failure (via the PP API) or safely make progress (with a bit of transactional state).

What happens with the authorization to the credit card processor times out? You don't know if it succeeded or not. You have to have a way to ask the credit card processor for "did this charge succeed?" and then process the payment if they respond with "no".

Don't you store the result of the authorization in your persistent storage? If you do, you can check whether you need to do the authorization again.

Credit card authorization might be better suited to a synchronous request, versus going through a message queue.

It doesn't really matter either way. Unless you are using distributed transaction protocols, you pretty much MUST store-and-forward and have idempotent processes in the consumer/receiver to guarantee something happens. Usually the easiest way is to generate a transaction ID, such as a UUID, at the very start and use it end-to-end as indicated by the earlier nonce comment. Then each step along the way can either locally dedupe, or refer to a shared database/API with it.

Well one difference is being able to easily propagate failures end to end.

Agreed about idempotent processes in the "consumer" processes. But I generally prefer marking an end user session in a persistent store before propagating the request for payment (effectively acting as a sticky current transaction ID), and making synchronous requests down the chain versus generating a transaction ID and doing a bunch of async message queuing.

It's just easier to know when something didn't work and display the appropriate user feedback. But your point is valid.

A lot of sensor data is like this. Missing a reading is not ideal but not the end of the world.

Reply on news.ycombinator.com