Performance Testing HTTP/1.1 vs. HTTP/2 vs. HTTP/2 and Server Push for REST APIs

I really miss old Opera which correctly implemented HTTP pipelining[1] :/

[1] https://en.wikipedia.org/wiki/HTTP_pipelining

Edit: article also doesn't mention about keep-alive, in fact it explicitly states that HTTP/1.1 opens one connection per request which is not true. Makes me think that the HTTP/1.1 demo disables KA to make effect even more dramatic.

Edit2: demos don't make actual network requests.

I call BS on the benchmarks AND the theoretical analysis. Every time I read those HTTP/X benchmarks, people don't mention TCP's congestion control and seem to just ignore it in their analysis. Well, you can't. At least if you call "realistic conditions". Congestion control will introduce additional round trips, depending on your link parameters (bandwidth, latency, OS configuration like initcwnd etc.) and limit your bandwidth on the transport layer. And based on your link parameter, 6 parallel TCP connections might achieve a higher bandwidth on cold start because the window scaling in tcp slow start is superior than in a single tcp connection used by HTTP/2.

Additionally, the most common error people do while benchmarking (and i assume the author did too) is to ignore the congestion control's caching of the cwnd in the tcp stack of your OS. That is, once the cwnd is raised from usually 10 mss (~14,6kb for most OSes and setups), your OS will cache the value and re-use the larger cwnd as initcwnd when you open a new tcp socket to the same host. So if you do 100 benchmark runs, you will have one "real" run, and 99 will re-use the cwnd cache and produce non-realistic results. Given the author didn't mention tcp congestion control at all and didn't mention any tweaks to the tcp metrics cache (you can disable or reset it through changing /proc/sys/net/ipv4/tcp_no_metrics_save on linux) I assume all the measured numbers are BS.

Out of curiosity, I implemented the same using websockets. You'll obviously need a real implementation of a request-response interface, but this will do for the test:

    // server
    ws.on('message', msg => {
      switch (msg) {
        case 'collection':
          return ws.send([...generate 500 links ...])
        case 'item':
          return ws.send(...generate item...)
      }
    })

    //client
    ws.send('collection');
    ws.onmessage = e => {
      const msg = JSON.parse(e.data)
      switch (msg.type) {
        case 'collection':
          collection.forEach(item => ws.send('item'))
        case 'item':
          items.push(item);
      }
    }

It finishes in 140-150ms, around the same time as the best HTTP version. That's without any caching and the client requesting each of the 500 items, when in fact the server could just push them. I'm surprised WS is not used more often for data exchange.

EDIT: actually ~50ms after removing the console.log() I added for each message...

Fascinating work, thank you!

Possible issue: have I misread it, or is the explanation of HTTP/3's advantages entirely missing from this section[1]? It's in the section header, but HTTP/3 isn't mentioned again until "The perfect world"[2] and that section doesn't clarify how HTTP/3 matters to the discussion.

[1] https://evertpot.com/h2-parallelism/#http2-and-http3

[2] https://evertpot.com/h2-parallelism/#the-perfect-world

"Compound requests are by far the fastest. This indicates that my original guess was wrong. Even when caching comes into play, it still can’t beat just re-sending the entire collection again in a single compounded response."

TCP is heavily optimized for shoving a single stream of bits across the wire. The other options have more overhead from multiple requests.

Many of my clients are behind MITM proxies that force a downgrade to 1.1, so I end up having to optimize for 1.1 and minimize total requests anyway. We've started down the path of GraphQL to aggregate requests, but it feels so counter productive (with respect to performance/effort).

I think the most interesting thing about this test is that it shows HTTP/2 isn't always faster than 1.1 --- in the first test, h1 compound is just as fast as h2, and in the second, even slightly faster. Considering all the additional complexity that HTTP/2 introduces, that doesn't seem like any improvement.

Also, "relative speeds" should really be "relative times" because otherwise the percentages imply the exact opposite. I was a little confused for a bit by that.

One of my favourite demos is https://http2.akamai.com/demo

Very interesting article! I wonder how compound requests scale as body size increases. Would we see individual HTTP2 requests become faster after items become "large enough"?

The thing that jumps out at me about HTTP/2: even if your browser can send N parallel requests for different entities, you're still making N database queries and incurring N request overheads.

This means that every layer of your stack has to be reorganized and optimized around HTTP/2 traffic patterns...or you can just batch requests as before, and save that overhead. It makes me think that the N entities problem isn't actually the most promising use case for HTTP/2...

Is there such a thing as an "HTTP connnection". Maybe the author means "TCP connection"

Maybe "Quick UDP Internet Connection" (QUIC) in the case of HTTP/2

QUIC appeared after the publication of another UDP-based, congestion-controlled protocol, CurveCP and used ECC by the author of CurveCP

QUIC seems tied to HTTP, whereas CurveCP, initially used for DNS, is application-agnostic

What are some other applications besides web browsers and web servers that are using QUIC

Story told well, comprehensible. Thanks!

Possibly so obvious it's not worth pointing out, but client side caching is also important for server side scaling. If your clients cache X% of responses, you can generally get away with ~X% fewer API servers.

So even if client performance doesn't improve much you should probably still cache.

If you care about performance, gRPC/Protobufs would be something to look at and compare against.

Maybe I missed it in the article, but is gzip-compression enabled?

If so, the total amount of data transmitted will be quite different when a compounded response is compared to individual ones.

Great article; the demos were very intuitive.

What about alternatives to HTTP entirely? Why not rely on lower level protocols/abstractions if performance is such a concern now?

Has the performance gain that bring http3 on mainstream workloads been quantified/benchmarcked?

neat work! didn't notice it mentioned in the article but, would there be a processor load difference between the two?