What size should I use for my queue

> My current thinking is that queues don’t increase average throughput. Instead, they act as buffers that absorb short-term bursts and timing differences between senders and receivers

Absorbing bursts is one purpose for a queue, but often not the only or even most important one. Other reasons include:

Correctness reasons:

* Providing a durable history of requests that were sent

* Creating at-least-once semantics for important (e.g. financial) data

Scaling reasons:

* Allowing a shuffle between producer and consumer to change shard/key affinity (can be done with direct RPCs, but would need extra library support)

* Pipelining variable-cost requests so that consumers can self-rate-limit and stay optimally loaded

So consider a setup where System A sends messages via a queue to System B.

Let’s say that the maximum rate of System A producing messages is 100 messages/second and for System B let’s make it 20 m/s for consuming messages. In this case it would take System A running at 1/5th capacity to saturate System B’s processing. In other words, the average rate here isn’t really governed by the average rate of queueing/dequeueing messages in a production system but by measuring the performance of each system separately and using the slower.

But this also highly depends on the type of systems you are building. For example if you are queueing log lines to be sent to the system logger in an application that only logs things occasionally but when it does there are a lot of them then you don’t really care about latency but you do really care to not lose any log lines so you make your queue as big as you can and catch up on the logs when there’s a break. In this queue you want to block when the queue is full.

On the other hand if your producer is a GPS-tracked robot driving around sending its position to a telemetry server or a thermometer reporting its temperature and this happens very frequently, then you can make your queue small and just drop the oldest or newest or random element in it. Latest update matters more than a full history as long as updates are frequent enough. Here you would size the queue based on your packet drop rate and delivery latency but also by the resolution of your GPS receiver or thermometer and expected rate of change. No point in sending a GPS update at 1000Hz since you’d have to be moving incredibly fast for each data point to not be just noise.

Sometimes a queue is also a way of helping the consumer batch some of the work. For example if the message first needs to be parsed and then processed as two steps then the event loop could fetch N messages and then parse them all at once, then pass each to a processor. By having multiple messages in a queue you are increasing the efficiency of this type of consumer, which should affect your batching strategy.

Optimizing for the network MTU when sending a bunch of messages at once could send them all as one IP packet (especially if using IPv6 which disallows fragmentation). So here you empty your queue as soon as it reaches a certain size and only read off a specific number of elements to keep the payload under a certain size. This way a queue acts as a buffer that is emptied at once. A queue with a high/low watermark could be used for this.

And queues are also sometimes very useful as synchronization mechanisms. Imagine multiple producers writing to a queue but the queue has a write lock so strict ordering of messages can be enforced. In this case whichever producer wrote their message first has its message processed first. This is very useful for serializing transactions that modify the same bit of data. But the size of the queue here could be completely independent of latency: you want to log that Bob spent $7.32 at Starbucks in the ledger of his bank account and not lose that write even if it gets processed days later. Yes Bob wants to get his coffee but at some point it is better to say that his payment was approved and hit him with an overdraft fee later than to wait on a long transaction backlog to clear.