There's a lot of reasons I expected to see as their justification, but "our application can't handle concurrent requests" wasn't exactly one of them.
Designing infrastructure for concurrent requests is definitely not. I've worked on shared hosting systems with high concurrency requirements and it definitely was more complicated than just installing an Apache MPM—we had to think about balancing load across multiple servers, whether virtualizing bare-metal machines into multiple VMs was worthwhile (in our case it was for a very site-specific reason), how many workers to run on each VM, how much memory we should expect to use for OS caching vs. application memory, how to trade off concurrent access to the same page vs. concurrent access to different pages vs. cold start of entirely new pages, whether dependencies like auth backends or SQL databases could handle concurrency and how much we needed to cache those, etc. At the end of the day you have a finite number of CPUs, a finite network pipe, and a finite amount of RAM. You can throw more money at many of these problems (although often not a SQL database) but you generally have a finite amount of money too.
I would be surprised if most people had the infrastructure to handle significantly increased concurrency, even at the same throughput, as their current load. It's not a sensible thing to invest infrastructure budget into, most of the time.
(You can, of course, solve this by developing software to actively limit concurrency. That's not a given for exactly the reasons that developing for concurrency is a given, and it sounds like Lucidchart didn't have that software and determined that switching back to HTTP/1.1 was as good as writing that software.)
Yes, most such cases should be rearchitected to not go through a single choke point. But my claim is that this isn't automatic merely by developing for the web, and going through a CP database system is a pretty standard choice for good reason.
In seriousness, though, I'm both curious and a little bit skeptical of what user experience benefit that architecture would give over a server-side request queue and a single worker against the queue. That would allow you to pay the cost of networking for the next request while the mainframe is working. You could even separate the submission of jobs from collecting the result so that a disconnected client could resume waiting for a response. Anyway, I'm not saying you needed all that to have a well-functioning system, I'm just not convinced that a single threaded architecture is ever actually good for the user unless it gives a marked reduction in overhead.
If you do expect the time to process requests to be multiple minutes in some cases, then you absolutely need a queue and some API for polling a request object to see if it's done yet. If you think that a request time over 30 seconds (including waiting for previous requests in flight) is a sign that something is broken, IMO user experience is improved if you spend engineering effort on making those things more resilient than building more distributed components that could themselves fail or at least make it harder to figure out where things broke.
Migrating to HTTP/2 delivers business value.
Responsive servers are more satisfying to customers than slow servers. That has business value.
Improving tangible metrics is not done to satisfy your emotions, it's done because of engineering rigor.
But to suddenly hit 50 for 1 second, then nothing for 9 seconds, well, that’s a tough spot to be in.
There must be some hard to find sequencing happening there, that they were not really exposed to before.
Presumably that's not the case for you?
This is an extremely common issue with Apache configurations, which often default to accepting hundreds of simultaneous requests without regard for memory capacity. If peak demand causes enough workers to spin up that Apache starts swapping, the entire server effectively goes offline.
Depending on the specific characteristics of the application, this could occur when load increases from 50 concurrent requests to 51 concurrent requests, or from 200 to 201, or from any integer A to B where A was fine but B causes the server to become unresponsive.
Saying that their A is 1 seems unnecessarily dour, given how common this problem has been over the past couple decades due to Apache's defaults alone.
"API server not fast enough, how do we fix it?"
"More threads! More connections!"
The problem, of course, is that HTTP2 is behaves like having infinite connections, so the "more threads" on the server are almost always detrimental to performance.
Less is more is the mantra I have unsuccessfully tried to drill. If your API (assuming a basic rest like service) is running at 100% cpu utilization, you've likely over provisioned it.
For example, 200 threads with 200 connections to a single service is insane and likely causing you to be slow already. Increasing that will negatively impact performance.
Going from 16 -> 32? That's more reasonable.
The actual post seems perfectly reasonable though (essentially “you might think you can just turn on HTTP/2 as a drop in on your load balancer a but if your server code hasn’t been written to rapidly handle the quick bursts of requests that enable HTTP/2 to provide faster overall loads to the client then this can cause issues; you should test first and make sure your server systems are able to handle HTTP/2 request patterns.)
I appreciate when people share war stories; I like to think that wisdom is knowledge survived.
A typical non-tuned Rails deployment, for instance, is gonna have queueing built in, with really not as much concurrency as one would want (enough to actually fully utilize the host; the opposite problem). So I'm guessing you aren't on Rails. :)
Curious what you are on, if you're willing to share, for how it effects concurrency defaults and options and affordances.
(I know full well that properly configuring/tuning this kind of concurrency is not trivial or one-size-fits all. And I am not at all surprised that http/2 changed the parameters disastrously, and appreciate your warning to pay attention to it. I think those who are thinking "it shouldn't matter" are under-experienced or misinformed.)
Sure. We use the Scala Play framework (https://www.playframework.com/). And it does have some queuing built into, but we have tweaked it to meet certain application needs.
Even then you would be handling more requests in parallel than the number of cores you have, but your concurrency would be limited by the cost of context switching and your memory capacity (having to allocate a sizable stack for each thread in most threading implementations).
Queueing is usually required for a stable multi-threaded server, but if you were doing async I/O you wouldn't need it. The extra memory overhead for each extra concurrent request (by means of lightweight coroutine stacks, callbacks or state machines) is not much different from the size it would take on the queue, and there is no preemptive context switching.
In most cases, you'll get the same behavior as having a queue here. Cooperative task-switching happens only on async I/O boundaries, so if you're processing a request that requires some CPU-heavy work, your application would just hog a core until it completes the request and then move to the next one.
It is not so easy. The article said it timed out (on the client).
When the queue is on client, the client start the timer when the request start.
That speed observed by clients should come from somewhere. In their case, they did not have a large reserve of performance to tap.
the problem is not making the pizzas in time but trying to get all pizzas started at once when there is not enough table space to even roll out that much dough, and then trying to squeeze all the pizzas into the oven at once, whereby several of them got messed up.
The logic here is not dissimilar at all: if the backend has no ability to queue and prioritise the requests, then the same function needs to be done elsewhere to safeguard quality of service.
> all existing applications can be delivered without modification....
> The only observable differences will be improved performance and availability of new capabilities...
Lucidcharts may have an inadequate backend, but it wasn't a problem until they moved to HTTP/2, so those statements weren't true for them. For anyone else rolling out HTTP/2, that is worth bearing in mind.
[1]: https://developers.google.com/web/fundamentals/performance/h...
This is only true when you look at a single client. If you look at a larger number of clients accessing the service at the same time, you would expect similar numbers of concurrent requests on HTTP/2 as on HTTP/1.1. Clients send larger numbers of requests at the same time, but they are done sending requests earlier so there are requests from fewer clients being processed concurrently. It should average out.
If you have, say, a 1000 clients accessing your service in one minute, I doubt the number of requests/second would be very different between both protocol versions. It would only be an issue if the service was built with a small number of concurrent users in mind.
But it didn't, unless you're saying that Lucidchart made an incorrect analysis. Is that your argument?
>Clients send larger numbers of requests at the same time, but they are done sending requests earlier so there are requests from fewer clients being processed concurrently. It should average out.
Again, it didn't average out. And you assume it 'will average out' at your peril. Maybe it will, maybe it won't. Lucidchart engineers thought that too and it turns out that was wrong in a way that wasn't foreseen.
>It would only be an issue if the service was built with a small number of concurrent users in mind.
I doubt Lucidchart 'was built with a small number of concurrent users in mind'.
This comment suggests otherwise: https://news.ycombinator.com/item?id=19722637
Under HTTP/1.1 requests may have been hitting the LB and then being scattered across a dozen machines. Each of those machines was in a position to respond on their own time scale. Some requests would get back quickly, others slowly, but still actively being handled.
Under HTTP/2 with multiplexing, if the LB isn't set up to handle it (and they often aren't) they can be hitting the LB and _all_ ending up on a single machine, which is trying to process them while some of those requests might be requiring more significant processor resources, dragging the response rate for all the requests down simultaneously.
According to the author “we do load balance by request, not connections and we do not use sticky sessions.” (Source: https://news.ycombinator.com/item?id=19722637)
The change in traffic patterns http/2 imposes was.
Hence the blog post.
Most would not think about the fact that your spikiness could increase 20x.