Hacker News Reader

Comment by orf | Hacker News Reader

No they shouldn’t.

The way I see it is that there are two opposing things you can optimise for that depend on queue depth: utilisation or latency.

If you care about processing each message as quickly as possible then queues should be empty. This often requires a higher cost and lower utilisation as you inevitably have idle workers waiting for new messages.

If you care about utilisation, then you never want your queue to be empty while something is polling it. This might be some background task that runs on a GPU - every second that sits idle is wasted cash, and those tasks usually benefit heavily from batching inputs together.

In this case you want it to always have something to read from the queue and shut it down the moment this isn’t the case.

View on HN

If a queue isn't approaching empty, then its input is exceeding its output. Therefore it's size is trending to infinity.

Since we don't have infinite space, we can expect eventually to lose some messages in this scenario.

I think this is a valid concern for Future Us to worry about. If you have alerts for when this is going to happen, you can punt the issue until it starts to become a concern and who knows if your product will be alive when that issue happens.

M/M/1 Queue theory doesn't apply to real queues... Because the work input into the queue usually does depend on the latency of previous work completed by the worker.

If there is a 1 hour queue for the checkout at a grocery store, you probably won't join the queue - you'll go to a different store...

In many systems you get the reverse. A client sends a retry without the ability to cancel the first request, now it has two requests in the queue.

Back pressure and circuit breakers matter.

> Since we don't have infinite space, we can expect eventually to lose some messages in this scenario.

Sure, that's technically correct but applies to basically everything. It is very likely your users/orders/whatever table in your database is also "trending to infinity" over time, except this doesn't mean databases should always be empty or that we should expect to eventually lose some users/orders/whatever.

Or, more succinctly, if your queue capacity is 10 million messages and your queue messages represent "houses purchased through our website", then in reality your queue capacity is infinite because nobody will ever purchase 10 million homes through your application per small unit of time.

Your response is correct, but there's a more fundamental flaw in the preceding comment: although both paragraphs are true in isolation, the implication between them is false. "The queue is _currently_ trending to infinity" does not imply "The queue will _continue_ to monotonically trend to infinity". Periodicity is commonplace in real-life situations; and while parts of Queueing Theory produces some insightful and helpful insights by ignoring it, care should be taken when applying them.

The growth rate of a queue is not constant, it could get bigger during the day, but smaller at night, since customers are not using the service.

If you're just making people wait 12 hours, or five minutes, or whatever - you aren't saving on compute. You're just adding a buffer. Maybe that is what you want, but it should be intentional.

This is usually the approach of people with fixed compute. If I have a VPS then I'll let it churn through the backlog overnight.

But this is also where the disagreement comes in with flexible serverless workers. They generally do not cost more, so you'll end up spending $24 in an hour rather than $1 an hour all day. If the serverless costs more then you are likely not working through your backlog.

I agree it should be a business decision, so you catch up on weekends instead for example, but non-technical folks don't always grasp those numbers and that they are getting a lag with their data.

It is a little counterintuitive, but "churn through overnight" still means you're decreasing the queue size. You're right that the cost really ends up being roughly the same, unless your service gives you a discount for compute overnight or something.

Completely agree that the queue still goes down, I just see comments on here about folks preferring a VPS because they run it 24/7 for the same price. That is probably fine for their scenario but the non-technical members probably don't understand that they aren't saving money by doing it this method but are adding a delay. That is my only point.

Exactly. I think a lot of people in this thread don't understand they "aren't saving money by doing it this method but are adding a delay" with their complicated auto scaling worker setup lol

> If a queue isn't approaching empty, then its input is exceeding its output.

It depends what you mean by "approaching empty". If you mean "empty most of the time", then no, that's wasted resources. If you mean "its size is decreasing most of the time", then no, not possible. Even if you mean "it is empty some of the time", then maybe, but that is not a strong requirement either.

Of course more average input than average output is bad, but stating it as "should be empty" or "should be approaching empty" seem to suggest the wrong thing entirely. It should be smaller than the target latency of the process.

Eh? It is possible for size to be decreasing most of the time. Just make traffic extremely bursty - add 60 on the hour every hour, and consume at one per minute.

What does average mean there? Increasing and decreasing are punctual events. If you mean the 1-minute average can be negative most of the time, that is true, but it would not work for other time horizons, whether larger or shorter.

the # of items in a queue is either growing or shrinking, there isn't the concept of "staying the same" and as you approach the line from shrinking to growing, performance (queue time) degrades after roughly 70-75% utilization.

Of course it can stay the same? Where the hell do you get this 70-75% figure from? This makes no sense.

Unless you have something outside of the queue monitoring the size and limiting items being entered into the queue, a queue is always either growing or shrinking. See: https://blog.danslimmon.com/2016/08/26/the-most-important-th...

70-75% is just a rough number, the real number depends highly on several factors related to your specific use of the queue.

It may not make sense but it is the reality.

You can only assert that it's "shrinking" or "growing" over a given period of time. It may have been shrinking over the last 5min while at the same time it has grown over the last hour. For a time window small enough, it has not changed, because getting 1 input or 1 output is a punctual event. For a time window large enough, it has necessarily grown, if it started empty.

So I disagree entirely with "it's either shrinking or growing". It is both growing, shrinking, and stable, at the same time, over different horizons, and I have no idea what law you are trying to state.

The link you provided above is a really good primer based on serious math, I don't understand how you think it supports your vague claims.

I think you should read up on this a little bit as your assumptions are not totally correct, and if you're using queues understanding this theory is pretty important and will save you some headaches.

Queues are either growing (adding items) or getting smaller (consuming items). There is not a state where they stay the same length, unless you're doing something really weird like monitoring to make sure there are always 10 items in the queue, or something, and adding if that number drops, but then you'd need a second queue-type structure to support that, and the conversation kind of goes off the rails.

What happens in between getting items or consuming them? Is that not a "state"?

I literally cannot understand how you think time works in your area.

It's a point in time, but it is irrelevant to the problem we're discussing and I'm not sure what value there is in discussing that static points in time exist. It's like discussing a runtime and saying "well, at this static point in time it takes zero seconds to run" - I just can't wrap my head around how it is relevant to the issue at hand.

You're also being really rude for no good reason. My polite suggestion is that you reflect on yourself here about why you are being rude on the internet to strangers. It could lead to growth and hopefully help you with whatever hurt you're experiencing.

The Little’s law[1] offers considerable guidance here. In the shortish term it’s safe to model the system as being stationary and then you need to provision for seasonality[2]. Upward or downward trending demand after correcting for seasonality indicates that the system isn’t stationary and your choices are either to scale up or reject requests. Nevertheless the optimal queue size is not necessarily zero except in cases where extremely low latency is necessary. The optimal queue capacity for most web service type applications is one that trends toward almost but not quite full when seasonal requests are higher.

[1] https://en.m.wikipedia.org/wiki/Little%27s_law

[2] broadly construed. For example a service that gets a mass of requests at the top of every hour displays seasonality.

This is so academic. Take any two samples; slope > 0 => we're going to be dropping messages!!! Slope < 0 our threads are idle!!!

> If a queue isn't approaching empty, then its input is exceeding its output. Therefore it's size is trending to infinity.

This is literally the slippery slope fallacy. You aren't accounting for minor fluctuations or finite timescales. If you were driving a car, you might say "if you aren't steering into oncoming traffic then you're steering towards the ditch" and conclude that no cars will ever safely reach their destination.

If the only valid state for a queue is empty, then why waste time implementing a queue?

GP said "approaching empty", meaning it should be empty in the long term. It's not a slippery slope.

It assumes the velocity of items being added to the queue never changes. In real life it speeds up (approaching infinite items), and it slows down (approaching empty). A more meaningful measurement would be something like average number of items in the queue and average time in the queue for any given item.

If you feel like you have enough spare capacity and any given item isn’t taking too long to process then it doesn’t matter if you are ever empty.

You're on the right track, but I personally find averages to be not a useful measure...on average.

Corner conditions are where I start to care:

* What is the worst case latency? How long until the work can be done? How long until the work can finish from the point at which it enters the queue?

* What is the worst case number of items in the queue? How large does the queue need to be?

* What is the maximum utilization over some time period with some granularity? How spiky is the workload? A queue is a rate-matching tool. If your input rate exactly equals your output rate at all times, you don't need a queue.

* What is the minimum utilization over some time period? Another view on how spiky the workload is.

Minimums and maximums I find are much more illustrative than averages or even mean-squared-errors. Minimums and maximums bound performance; an average doesn't tell you where your boundary conditions are.

In general you don't want your queue to fully fill up, but like the other poster said, it's some tradeoff between utilization and latency, and the two are diametrically opposed.

Sure, I guess by average I just meant shorthand for some sort of measurement for making decisions about resources allocated to your queues. Median, mean, P99, whatever is useful.

We use queues for various jobs, some of which are time-sensitive (e.g. emails), so the queue should be empty most of the time, some of which are background or re-processing (e.g. re-indexing), so queues can balloon and then take days to drain again.

The unifying aspects of using queues for us is that it allows us to load-balance jobs across workers, and allows us to monitor throughput in a centralised fashion. We can set alerts on the time-sensitive queues and then use different thresholds for the background queues, but we're using the same metric data and same alerting system.

No it isn't. It is math and queuing theory.

What about a queue that is the ingress point for a fan-in between O(N) active continuous producers to a single active continuous consumer?

In such a case, you'd expect at least N messages in the queue at all times — and often a small multiple of N. Not because the consumer can't consume those messages in a timely manner, but because, given a reliable queue, messages can't be consumed until they've been 2PCed into the queue, and that means that there are always going to be some messages sitting around in the "acknowledged to the producer, but not yet available to the consumer" state. Which still takes up resources of the queueing system, and so should still be modelled as the messages being in the queue.

This is true only when you have relatively steady flows, or infinite multiprocessing. In that case, yes, the queue is in practise either "almost empty" or "infinite".

I think the reason you get the responses you do is people are used to systems with high variation in flow rates that can only process a limited number of items at the same time, and for these, queues can soak up a lot of variability by growing long and then being worked off quickly again.

> should always be full

> always have something to read from the queue

In the first case, you've gone a bit far. To maximise utilisation, you want the queue to never be empty when polled.

Indeed, and this is what I meant but worded poorly.

Obviously you don’t want it to be 100% full, but you do want it to be the kind of full you get after a nice meal but you still have room for desert and a coffee.

Good reading here: https://blog.danslimmon.com/2016/08/26/the-most-important-th...

You really don't want your queue more than 70-80% utilized. Once it gets to 75% you should consider that "red zone" and you need to fix it somehow. Once the queue is 80% you're really hitting into some issues. Above 80% and you will have significant problems.

Thanks for this, that has helped me see something I missed.

If you’re dealing with something like SQS and are receiving variable sized batches of messages, the consumers can be classed as “100% utilised” (always processing a batch of messages with no empty receives) whilst the queue would not be considered 100% utilised (the average delivered batch size is always less than the max).

What issue are you trying to solve? The expense of polling?

Obviously there is a line where you do not want a bunch of expensive consumers hanging around - but this isn't the place IMO to micro-optimize your costs. You shouldn't have like 20x the consumers you need but you shouldn't worry about having EXACTLY the correct amount.

In your example, are you implying that if more jobs come in more workers get spawned? Is that specific to the Amazon service?

If a process is busy processing a task, it is busy.

A process could receive N tasks in a batch and process them at the same time.

In this scenario it is possible for all workers to be utilised whilst the queue is not “100% utilised” - which is a failure mode the link you shared explained:

> And by induction, we can show that a system that’s forever at 100% utilization will exceed any finite queue size you care to pick

The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks.

It is specific on the utilisation type though, I’m mainly thinking about ML inference that has a fairly fixed initial cost per inference but scales quite nicely with >1 input.

You can't have 100% utilization of any processing capacity (ignore queues) without reaching this type of failure. 100% CPU utilization is essentially just meaning you filled up the queue on your worker server.

> The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks.

Right, but at some point they will have to have SOME idle time. If they are NEVER idle the queue by definition is growing, not shrinking, as you said.

Unless you have some system that monitors queue length and doesn't add jobs if it is like >n, which is what the article suggests (maximum queue size). But then you have to figure out what you do with those jobs that can't be added, which probably means just creating some kind of queue higher up in the pipe.

If they are processing more than is being added, eventually things will run out of the queue.

that's a great paper that clearly explains how "back pressure" [my term] at high utilization causes latency to go hockey-stick. (And thanks for the link bc I apparently forgot to bookmark it when I initially read it.)

You're welcome! Important for people to get this I think bc we're using this tech more and more.

Isn't a full queue basically equivalent to no queue? The same number of requests is being dropped.

No, there are use cases where new entries are not ingested by requests - sometimes you know all work that needs to be done upfront. And sometimes you have a situation where you have a few requests coming in over time but that you at the same time need to sometimes go through everything you have seen earlier, once more and flush it through the queue.

Not all queues serve (soft) real time requests and not all queues are bounded (1). If the rate of incoming queue items is predictable but not constant (for example, average load in the weekend or at night tends to be lower for many systems) you can often get away with having enough capacity to serve the average load while using the queue as a buffer. This is fine if the work done is not time critical, like sending most types of marketing emails.

You might argue that a queue with many items in it which can still accept more items is not really "full", but it is clearly not empty either.

(1) Obviously all queues are bounded by the capacity of the hardware, but since most queueing software can persist to disk and/or lives in the cloud with elastic storage this is not usually a problem.

The difference is that the requests that are being served are getting extreme latency. You are serving the same number of requests, but every single one is X days old.

If your queue is always full (100% utilization) your jobs will on average never be processed.

Love the downvotes! People should read this: https://blog.danslimmon.com/2016/08/26/the-most-important-th...

I understand this is a little counterintuitive but it is true. You should not have a queue that always has a job ready to be processed.

Queue isn’t full. It isn’t empty and things are being processed not in “real” time. I’ve always run my queues where they are never empty. If queue depth hits certain levels, workers are scaled up. When it drops, they are scaled down. Last thing I want is under utilized instances.

Unsure what you're saying. If your queue is NEVER empty wait times will grow to infinity. You shouldn't aim for this. You should aim for something reasonable like, "empty 50% of the time" etc..

They haven’t been empty for two years and my waiting times average 5 minutes and 30 minutes during high load times. I don’t understand how they will grow to infinity.

> I don’t understand how they will grow to infinity.

You either have to be consuming faster than the queue is growing, or you are adding faster than you consume. There is no equilibrium.

If you are consuming faster than it is growing, at some point you'll consume all jobs, meaning your queue is sometimes empty. If you are adding faster than you consume, your wait time approaches infinity (tho in reality you'll fill up your queue storage first, of course).

That is only the case if the consumption rate is fixed. If the rate of consumption is adjusted in a feedback loop based on queue depth and/or latency then clearly you can keep your queues in a balance of being neither empty nor infinite.

No, it still applies with variable computing. All you're doing is adding a N length or X second buffer to your queue. You are just shifting your "zero" over by N items or X seconds, which is the time you wait until you scale up.

You may need that or want to add this buffer - but you're not solving any computation resource problem directly. If your jobs take XYZ CPU cycles they take XYZ CPU cycles - it doesn't matter if you make them wait a little bit.

You could even be spending more money, since it does take some resources and cost to scale things.

It doesn't mean it's not valid to do things this way, I'm just saying it's still the exact same problem.

Your comment only shows that if you value resource utilization, you shouldn't use queues.

Because, no, nothing you said changes the fact that queues only do anything when kept mostly empty. And the only thing they do is some short term trade of latency for reliability. They actually can't change utilization.

I’m currently waiting for a flight. Isn’t airport is a kind of queue that buffers people over time then delivers them to the sky at steady, fixed intervals?

This airport seems busy but the planes seem full, how can we improve utilisation?

What does it mean to improve utilization if the aircraft are at 100% utilization?

Bigger aircraft I guess. But the point is that we are at full utilisation despite the “queue” (airport) never being empty, and thousands of people awaiting “delivery”.

Which refutes the original point about queues not doing much unless they are empty and being unable to effect utilisation. If you have a somewhat inelastic output that benefits from batching, but has variable inputs, they are absolutely crucial to utilisation.

I'm being nitpicky but my intended prompt was to get you to define what needs to be improved, which you did. You need to improve (or increase) capacity, not utilization. The two are related, but they aren't the same. Utilization being at 100% means it only has two ways to go: Stay at 100% or drop below 100%. Which one constitutes an improvement? It depends, but for the airline utilization at 100% is already what they want (by the numbers, should be their maximum profit) so it can't be improved. Only capacity can be improved.

Capacity of course has a cost. One of the benefits in computing is we can easily up and downshift Capacity based on this like queue latency.

as you inevitably have idle workers waiting for new messages.

This is no longer the case using the latest in serverless IaaS architecture, where the pool of queue workers dynamically expand to meet demand.

This has always been a thing, far far before the “latest in IaaS architecture”. The idle polling still exists except they manage it for you. Their margin is then improved if they intelligently reduce wasted polls through a variety of techniques.

However, yes, you don’t pay for this or care about it much. It’s good.

Some server will still be idle and you'll pay for it. Dedicated hardware utilized close to 100% will be massively cheaper than IaaS.

Seems like there's some implicit assumption in this comment that everything is rented by the minute and latency never matters. Renting things by the minute isn't really the cheapest way to buy lots of things though. Also, its sometimes nice to respond to the content of the article instead of its title.

I think the implicit assumption in the comment is that it would be possible for readers to understand the underlying idea of the comment rather than only see the specific example. “resource cost per minute” is not the only thing helped by a non-empty queue.

Also please don’t accuse someone of not “responding to the content of the article” when the content is a bog standard list. If you want my response to that: “yes, that is indeed a list of 4 things queues help with”. Not sure it adds much to the discussion.

When will my users be happier because I took longer to do things on their behalf? I apologize for not understanding the unwritten portion of the comment.

I guess I should really ask, why do people care about utilization instead of throughput, latency, and cost? We could increase utilization by rewriting everything to be more wasteful, for example, but it doesn't seem like we should do it.

Utilization is choosing throughout and cost over latency.

People care about utilization when, eg, processing petabytes of data — something that takes hours to days, where you’re optimizing batch throughout and cost. By making sure you efficiently utilize your hardware.

> Utilization is choosing throughout and cost over latency.

For a given rate of incoming work, increasing throughput decreases utilization. So I guess it's not choosing throughput too much.

> People care about utilization when, eg, processing petabytes of data — something that takes hours to days, where you’re optimizing batch throughout and cost. By making sure you efficiently utilize your hardware.

If you have fixed resources and lots of work, it would be good to have nonempty queues to prevent them from idling. That is maybe the big unwritten part of the comment upthread. It doesn't seem worth saying "no, the article is wrong in all cases because of this particular case!" though, so I'm not sure.

If you define throughput as "processed items over time per processor", the tradeoff becomes clear. Given the rate of incoming requests, you can get more processors, which will increase cost and reduce latency and throughput-per-processor, or you can get fewer processors, which will decrease cost and increase latency and throughput-per-processor.

If you have fixed resources and fixed work, wanting your queues to be empty is a wish, not a strategy.

I came here to say the same and saw your comment. Totally agree.

You’re assuming the consumer is the one producing batches.

Wrong. They should.

Any queue that fills up, say halfway, is an indication the system is in a precarious state.

Its simple. Queues have limited space if the velocity of data going in is faster then data going out the queue will fill up until it's full.

A half full queue indicates the velocity is in such a state and such a state is unsustainable. You need to build your system such that it stays out of unsustainable states as much as possible.

If you're finding that your queue fills up half way a lot that means you're building your system in such a way that it operates at the border of sustainability. In other words the system often ingests data at rates that cannot be sustained indefinitely.

If a system with a queue half full operates in that same state twice as long, you now have a full queue and you're losing data.

In essence your queues should mostly be empty most of the time. At best you can account for occasional spikes as the article stated but a queue even half full should never be the operational norm.

This response lacks nuance and focuses too much on the operational side of queues.

Not all queues are “realistically bounded”. Let’s take a hypothetical silly “order queue” where each entry represents a €10 item that’s been purchased. The queue feeds into a boxing machine that puts the item in a box and posts it to the user.

The point at which a queue hits an upper bound would represent billions of euros of orders. That’s not really a realistic failure mode to hit without noticing long beforehand.

Meanwhile, if you always want your queue empty then your boxing machine is going to be shipping thousands of small boxes because the moment an order arrives, it’s processed into a box.

It’s much more efficient to have a small, managed and time-bounded backlog of orders that the boxing machine uses to batch orders together in a smaller number of larger boxes.

Wrong. Your thinking is where the nuance is lacking. As I said you need to think of the system in terms of it's derivative. The rate of data consumed must on average be faster then the production. The net difference of these rates must equal a negative velocity with the rate of data going in as a positive number and data consumed as negative.

There is no system on the face of the earth that can operate at an average positive velocity. That is categorical failure.

This is mathematically and logically true no matter how big your queue is.

Even for the situation your describing... A really big ass queue, the average velocity of that system must still be negative. Unless your queue is so big that it can withstand an average negative velocity over the lifetime of the system which is kind of absurd.

Most of the time your queues should be empty. If your queues are observationally often filled with data but not full it means you're still operating at a negative velocity but you've tuned your system to operate at the border of failure.

It means your system hits unsustainable velocities often. You may want this for cost reasons but but such a system is not robust enough for my comfort. The queue should be empty most of the time with occasional spikes at most.

This is a very long-winded way of saying “you should have enough capacity to process all items in the queue before the queue fills up and everything explodes”.

I agree. But that’s not the argument the post is making, nor does it support the position that “queues should be empty”. No, they shouldn’t. It’s fine to let a queue build up (and thus not be empty), then process all the items in the queue in a short time frame. There are efficiency savings if you are able to do this.

No. That is not what I'm saying. I'm saying on average queues must be empty most of the time. And if you are seeing queues that are filled most of the time, your entire system is operating on the border of failure.

This is true even when you have a big ass queue. The system must on average be more empty then it is filled.

This is not to say that your system has to never have spikes in data. Your system can often have spikes, but this cannot ever be the norm as in the amount of spikes in data production rate cannot push the average production rate to exceed the average consumption rate.

I don’t see how this holds true. Let’s say I have a process that takes 1 hour to process a batch of items up to 1000 (I.e doesn’t scale linearly with input size). So, 1 hour to process 1 item and the same 1 hour to process 1000 items.

If I have 1 message per minute being added to the queue, I can safely run 2 of these batch processes once a day to clear the queue. Or more generally can launch (queue_size / 1000) tasks once a day.

Because surely it doesn’t matter about the queue being empty, what matters is the rate at which messages can be processed.

With 2 processes running once a day the rate would be 2000/24/60 = 1.3 messages per minute in aggregate.

The queue can be filled 23 hours per day without issue and without being on the brink of a failure state.

Edit: you edited your comment whilst I was replying to specifically mention consumption rate. The aggregate consumption rate is all that matters, but it doesn’t follow that the queue “has to be more empty than not empty”. Those are different issues.

You're talking about a different scenario. I'm talking about continuous inflow and outflow. Where outflow is processing (fixed rate) and inflow is traffic (variable). This is the classic assumption. In this state the space used in the queue is an indicator of mismatched consumption and production velocities.

You're talking about a single batch job that runs once a day and empties the queue at the same rate no matter how much data is in the queue. Thus the more data you grab in batch the more efficient the processing so you grab data once per day instead of continuosly. This is a clear exception from the assumed norm that needs to be mentioned.

That is not to say that your case is invalid. Many analytic databases possess your ingestion parameters. But the case is exceptional enough that if not explicitly mentioned it can be assumed that such a batch job is not a primitive in the model being discussed.

It's not really that exceptional. You can do similar things with e.g. batch flushes every 5 ms for synchronous web requests and the same ideas will apply. This can give a massive increase in throughput while possibly decreasing overall response time for OLTP.

It's common, but exceptional enough that it should be mentioned because it is a specialized database that does batch ingestion.

Let's say you flew from one city to another. People assume you took a plane. If you took a helicopter, though helicopters are common, it should be mentioned otherwise it won't be assumed.

It doesn't take a special database though. You can do this sort of thing in your application in like 10-20 lines of code (assuming you have a framework with queues and promises). It works great with mysql/postgresql which both benefit dramatically from batching queries.

The context here is external queues like kafka. If you're writing stuff to a queue in your web app then it must be a relatively small load. This is different.

For analytics databases the batch ingestion requirement are usually so big that you have to save it to the file system hence the need for an external queue.

Heap allocated queues are different, at that stage I would say that it's in "processing" already and popped out of the external queue.

I guess "relatively small load" is... relative, but I've written that kind of thing to be able to handle ~70k sustained OLTP requests per second (including persistence to postgres) when load testing locally on my laptop. In any case the same thing applies to external queues. Your workers will often be more efficient if you pull off chunks of work to process together.

By small load I mean size. The frames in your batch jobs are tiny and few in number if you're getting 70k batch jobs per second.

It's more similar to streaming... what you're doing here. In that case my velocity measurements are more applicable. You want your queues to be empty in general. A batch job is run every hour or something like that which is not what you're doing here.

If you ran your load test for 5 minutes and you see your queues are 50 percent full. Well that means 10 minutes in you'll hit OOM. Assuming your load tests are at a constant rate.

If your queues are mostly empty then it can handle the load you gave it and have room for spikes. It's just math.

I disagree it’s exceptional even if my examples have been caricatures.

Very few technical problems don’t have any elasticity in outflows, but a lot of problems do benefit from a form of batching. Heck, even something as simple as inserting records into a non-analytic database benefits greatly from batching (inserting N rows rather than 1) and thus reducing TPS, or calling an external service, or writing files to some storage, or running inferences, or writing to a ledger, or anything.

Batching, in all forms, exists absolutely everywhere from super-scalar CPU uops to storage to networking to the physical world. The exception is to do something without batching, even if the whole stack you’re using from the ground up is built on it.

Given the common use case of batching, which inherently requires items to be queued for a period of time, how can you say that queues should always be empty? They would statistically rarely be empty. Which is a good thing. Because it enables batching.

If you want to queue and insert 1000 records into a database with 1000 transactions across 1000 workers, then say your workload is highly optimised because there are never any records in your queue, cool. Not very true though.

>Very few technical problems don’t have any elasticity in outflows,

They have elasticity but you don't care for it. You run it at the maximum. As you mentioned the only case where you wouldn't is if you have some transaction that doesn't scale linearly with with frames/events.

>Given the common use case of batching, which inherently requires items to be queued for a period of time, how can you say that queues should always be empty? They would statistically rarely be empty. Which is a good thing. Because it enables batching.

In my experience, while batching is not uncommon, streaming is the much more common use case. So we can agree to disagree here.

Queues can be full most of the time without the system being in a precarious state. Take for example an extremely spiky pattern of processing: once a day, a big batch of jobs is delivered. You know ahead of time how many items there are and have a pretty good handle on how long it takes to process one item. The items should be processed before the next batch arrives. You can then tune your system to empty the queue just before the next batch arrives. The queue is non-empty the majority of time, yet the system is perfectly stable.

Now, that is an extreme example, but similar behavior exists. It’s fine to have queues partially filled most of the time, as long as your average rate of processing exceeds the rate of incoming items by some margin.

AS I said in the other post I am talking about the average and most likely use case. a flat and continuous rate of data processing and varied rates of data production.

Spiking the processing is not what people think about with queues and is most likely NOT what the author of the article is talking about.

Typically there's no reason why you would spike your processing unless such processing doesn't scale linearly with the amount of items processed (as the author in a sibling post to yours mentioned). Such a case must be deliberately introduced into the discussion as an exception to the model because it's not what people think about when discussing this. Or maybe it is given that 3 people brought the same exact exception up. Welp if that's the case then I can only say the author of this article and I certainly aren't referring to this use case.

edit: I'm rate limited. So I'm referencing another post in my response that you won't see until the rate limit dies and I post it.

> You may want this for cost reasons but but such a system is not robust enough for my comfort

Your comfort is secondary to the business needs that the system is designed to fulfill. In practice some risks are better to accept than mitigate

My comfort is a measure of business aversion to risk. It depends on the business of our respective industries.

Either way you have admitted that you tuned the system to operate with an acceptance of risk. Likely for cost reasons. This is fine, if you deliberately chose to do this.

> This is fine, if you deliberately chose to do this.

Yet you bravely asserted that

> At best you can account for occasional spikes as the article stated but a queue even half full should never be the operational norm.

Suddenly this thing that you said "should never be" changed to "is fine"

When humans communicate. We don't talk about exceptions where you deliberately choose to operate outside of the norm. If you deliberately choose to drive a car without a seat belt knowing this risk. This is fine.

You want to sky dive while deliberately choosing not to use a parachute? That's fine. But don't expect me to adjust my communication to account for your exception. You chose to do this.

But let's not pretend that you don't know about this fact in human communication. It's an obvious thing you know. Yet you chose to characterize your post in a way that negatively makes my statement look unreasonable and your actions look normal. Please don't do this. If you choose to sky dive without a parachute, that's fine, but admit you are operating with risk for the sake of cost. Do not try mischaracterize my intentions, that is just rude.

> You need to build your system such that it stays out of unsustainable states as much as possible.

You assert that without justification. Typical counter-example would be system where you have predictable daily load-pattern, it can be perfectly reasonable to build the system that accumulates items during peak hours and clears queues during quiet hours.

You are making a somewhat subtle error here, which is assuming that the (output_rate-input_rate) is statistically independent of the size of the queue. Note that this does not necessarily imply that the output process changes based on the size of the queue. For example, consider a microphone that emits 100 raw samples to a buffer per ms and a real-time thread scheduled every 2ms that processes very quickly. a 300-sample buffer is more than enough here, even though the buffer will be over half full about a quarter of the time, and will never be empty. In this case, the velocity conditional on size being less than half is -100 samples/ms and the velocity conditional on size being greater than half is 400 samples/ms.

This example suffers from the same problem as the other counter examples. Why are you deliberately slowing down the processing to only happen at 2ms intervals? If you have continuous processing the queue will be empty always assuming processing is faster then data production.

You're maybe referring to what the other person mentioned... processing that does not scale with the amount of data processed so you want the processing to operate on the biggest batch of data possible. This is, while a common use case, not common enough that it should be assumed without be mentioned.

However, in your example it seems to be happening on the same computer (you mentioned mic), (and thus not web, which is what's the assumed context here) therefore data production rate doesn't seem to spike. If your processing is faster then data production you don't even need a queue, just stream each frame straight to processing in a single thread. But this also assumes the nature of your processing. The details for the nature of the processing can change things to require a queue for something like the averages of frames at discreet intervals... but that's different from the use case of queues like kafka or zmq for which I think is the main context here.

Edit: I reference other responses here, but you won't see them immediately. I'm currently rate limited and I have those replies in a queue so those will be posted later today.

Reply on news.ycombinator.com