Hacker News Reader

Comment by hiAndrewQuinn | Hacker News Reader

The maximum number of connections thing in HTTP/1 always makes me think of queuing theory, which gives surprising conclusions like how adding a single extra teller at a one-teller bank can cut wait times by 50 times, not just by 2.

However, I think the problem is the Poisson process isn't really the right process to assume. Most websites which would run afoul of the 2/6/8/etc connections being opened are probably trying to open up a lot of connections at the same time. That's very different from situations where only 1 new person arrives every 6 minutes on average, and 2 new people arriving within 1 second of each other is a considerably rarer event.

[1]: https://www.johndcook.com/blog/2008/10/21/what-happens-when-...

View on HN

And if memory serves if you care about minimizing latency you want all of your workers running an average of 60% occupied. (Which is also pretty close to when I saw P95 times dog-leg on the last cluster I worked on).

Queuing theory is really weird.

Why 60%? I suppose if they are less than 1% then latency will be even less.

Poisson could probably explain it elegantly, but he's not here, just us.

Most analyses I've read say the threshold is around the 80% mark [1], although it depends on how model the distribution, and there's nothing magical about the number. The main thing is to avoid getting close to 100%, because wait times go up exponentially as you get closer to the max.

Little's Law is fundamental to queueing theory, but there's also the less well-known Kingman's formula, which incorporates variability of arrival rate and task size [2].

[1] https://www.johndcook.com/blog/2009/01/30/server-utilization...

[2] https://taborsky.cz/posts/2021/kingman-formula/

If you look at the chart in the second link, where does the wait time leave the origin? Around 60%.

The first one is even worse; by 80% you're already seeing twice the delay of 70%.

If I were to describe the second chart I'd say 80% is when you start to get into trouble, not just noticing a slowdown.

I said minimize latency, not optimize latency.

Really both of those models show 60% as about the limit to where you're still effectively at the baseline for latency. 80% is just about the limit to where you're up there in the exponential rise, any higher and things become unusable.

0-60 and you're still at minimum latency. 60-80 you're at twice the latency but it's probably worth the cost savings of the extra compute density since it's still pretty low. Higher than 80 and things are already slowing down and getting exponentially worse by the request

dog food dog leg dog ram lol

dog leg is what people who aren't pretentious prats call an 'inflection point'.

What do people who aren't pretentious prats call "latency" and "quantiles"? =P

dog race and dog quarter

Wait time and maybe clusters?

First time I ever hear of this term, even though it's apparently applied to everything from pixel graphics to traffic engineering to rocket launches. I mean, you US-ians seem obsessed with dog legs to an unhealthy degree[0].

[0] - https://en.wikipedia.org/wiki/Dogleg

We love dogs. Don’t you?

I'm more of a cat person.

... "The cat-leg in this chart at 75 req/s suggests that we should set our autoscaling rules somewhere in this viscinity."

im more a free the slaves type of person

I've never heard dog leg before.

ngl inflection point sounds less of a pratty thing to say than dog leg lmao

Somehow I was building distributed systems earlier in my career before I learned about queuing theory and learned this the hard way.

Nowadays with DB stuff I tend to get assigned new infra leads who see a DB cluster at 50% CPU utilization and think they can go down two instances sizes without severely impacting latency.

For me it was seeing a machine at 70% utilization and thinking they could squeeze another small service onto the box. The first time it didn’t sound right, and after that I knew it was a bad idea. By the third time I was willing to throw a veto if they wouldn’t listen to reason.

And the thing is, even if you stuff a low priority service onto a bunch of boxes, and convince the OS to honor that priority fairly, the fact that the service runs reasonably at all gets baked in as an expectation. Maybe it’s the hedonic treadmill, one kid expects dessert with every meal because they’ve always gotten it, and another knows it’s a special occasion. But anything given is jealously guarded when you have to take it away. Even a “best effort” batch process that is supposed to finish on some interval, is missed when it no longer does. And somehow it’s always your fault.

I’m sure the grocery store employees who are assigned as backup tellers constantly get grief for not getting their other tasks done “on time”.

This is, by the way, one of those problems the Cloud solved, because management couldn’t use friction with the procurement and operations departments to try to pin you into oversubscribing machines. Cloud computing usually leaves the barn door open so that you can requisition another server and ask forgiveness later. And I think at some level developers all know this, so we went along with the dope deal. Because fuck Operations and Accounting for making us the scapegoats to problems they cause.

Any resources you’d recommend to learn more about this?

for context: 10 yrs experience as a software engineer and only a couple on high traffic products. Some things (like this) continue to surprise me.

Peep https://yzr95924.github.io/pdf/book/Basic-Queueing-Theory.pd... , honestly just being aware of it is already half the battle.

The more examples you read the easier you can recall this stuff. I find the single to two teller examples very compelling myself. How quickly a line can build up and how long it lasts when people/tasks are arriving at 90% of capacity and you get one burst of extra traffic is quite shocking the first time you understand the math.

Can't it cut wait times by infinity? For example, if the arrivals are at 1.1 per minute, and a teller processes 1 per minute.

You've forgotten about banker's hours :)

There was a discussion not tool long ago about modern banks still with archaic practices. I have accounts at two different banks, and if I make a transfer request before 1:45PT, it is counted as same day. That makes no damn sense to me why that's a limitation at all today. It's not like a human needs to look at it, but even so, why the 1:45PT cutoff? Is it because it is 4:45ET? Then why not list it as that? And why does a banking computer system care about timezones or bankers' hours at all. It's all just mind boggling lame

I know my father sometimes had to take 1 am phone calls because the insurance industry runs a lot of batch processing overnight, when the systems aren't competing with OLTP traffic. Banking software may be built the same way.

> Is it because it is 4:45ET? Then why not list it as that?

Because a lot of their customers are too stupid to understand timezones.

I guess that's saying more about the left coast customers then, right? as all of the customers from the other time zones have to do the conversion.

Well, if someone misunderstands and sends their payment by 1:45ET or 1:45CT, it's not a problem for the bank.

It’s never a problem for a bank, because they’re depend on late fees to hit their profit projections.

A late fee that a customer quietly pays is a win. A late fee that a customer feels you screwed them over on and kicks up a fuss about can be more trouble than it's worth.

If more businesses were based in Indiana we would understand this problem better.

Gary Indiana is a population center on Lake Michigan, across the Illinois/Indiana border from the Chicago exurbs. They’ve elected to be in the same time zone as Illinois, while the rest of the state is in the same time zone as Ohio. And 1997-2006 was just a clusterfuck where another county joined with Gary, and a bunch of counties decided not to follow DST.

I am not smart enough to know what time it is without asking a computer.

This is just a rude guess and not relevant to the actual reason

Rude perhaps, but I think it very much is the actual reason. "If they listed it as 4:45ET their support costs would be higher" is perhaps the more polite way of putting it, but the meaning is not actually any different.

I’m too stupid to understand time zones.

Because everyone is too stupid to understand them at 2 am when alarms are going off. All backends should always be in UTC, even if your boss tells you we will “never” move our servers to another time zone. Because they’re going to buy or be bought by a company based in Chicago or New York or find a vertical so fat they want a second office there, or both. And then they will not remember promising you that you would never have to do a time zone migration. Why are you being so grumpy, just fix it?

It's because when you do transfers, the banks will reconcile their accounts at the end of the day (e.g, if one bank deposits more to another, they will need to make up the difference with their own capital).

These cutoff means banks have certainty about the transaction, as these reconciliation is batched, rather than real time.

I think a better take on this may be credit cards. And an old classic article, “your coffee shop does not use two phase commit”

https://ieeexplore.ieee.org/document/1407829

Could be that, could also be the people taking a long time aren't at least causing a bottleneck (assuming there arent two of them at the same time). So you have this situation like this: first person takes 10 minutes, while there are 9 waiting in line that take only one minute a piece. With one teller, average wait time is ~15 minutes. With two tellers, its now ~5 minutes.

Which is why it is highly annoying when there's only one worker at the coffee stand, and there's always this one jerk at the front of the queue who orders a latte when you just want a coffee. With two workers, the people who just want coffee won't have to wait 15 minutes for the lattee people

And I've also noticed a social effect, when people wait a long time it seems to reinforce how they perceive the eventual serviced, that is, they want more out of the interaction, so take longer. Which makes it the situation even worse

> there's always this one jerk at the front of the queue

Here in the espresso world, that’s not so bad. But the ‘vanilla oat milk decaf, and also a hot muffin with butter’ is tedious.

There is a roaster in Auckland that’s been there since the ‘80s. On the counter it says ‘espresso, flat white or fuck off’. Clear and concise. I like it. https://millerscoffee.co.nz/

"On the counter it says ‘espresso, flat white or fuck off’"

Sounds a bit pretentious to me. I generally order a coffee, no milk ... ta.

Try ordering a “just a cup of coffee” in AU/NZ and they will look at you with a blank expression. Espresso is the norm there.

Presumably whoever wrote that sign anticipates this criticism

I'd order a "fuck off espresso" in that situation just to see what happens.

That’ll teach them to use Oxford commas.

Luckily, we don't get stuck behind someone using a check any more.

Instead we're stuck behind people trying to make their phones do something and taking PoS surveys...

The Poisson process is a random distribution. Just because its average is 1.1 per minute doesn't mean you can't have 2 people show up at virtually the same time, it's just pretty rare. Or there's like a 10^-100 chance a million people show up at the same second, etc etc. You'll always have some nonzero average queuing time if you calculate it across the whole distribution for that reason.

Reply on news.ycombinator.com