HPACK: the silent killer (feature) of HTTP/2 | Hacker News Reader

HPACK: the silent killer (feature) of HTTP/2

blog.cloudflare.com

102 pointsjgrahamc53 comments

This really seems like overkill to me. We've made HTTP/2 that much more complex to implement (a custom compression and dictionary not used anywhere else) to compress headers. And complexity always leads to fun new vulnerabilities. I guarantee at some point we'll see at least a DoS come out of this.

I would add that it also consumes more resources to compress these headers (I know huffman is fast; but it's not as fast as plaintext), but you can probably cache a good portion of it, and then just encode the per-connection values at the end. Although I'd be surprised if Cloudflare has nginx doing that.

Right now, my domain's HTTP header is 192 bytes:

    Content-Type: text/html; charset=utf-8
    Content-Length: 3884
    Connection: keep-alive
    Content-Security-Policy: upgrade-insecure-requests
    X-Content-Type-Options: nosniff
    X-Frame-Options: deny

And more than half of it is just optional security enhancements.

So now I'm going to compress that header so that it's ... what, 100 bytes? Even on my site that's incredibly lean and lacks images, it's around 20-30 KiB per page of content. Modern web pages now average being larger than the original DOOM video game (2301 KiB.)

So we're doing all of this to chase roughly a 0.05% reduction in bandwidth per page? We're worrying about the spot on the carpet while ignoring the elephant in the room.

From the article: It is difficult to see, but we get 1.4% savings in the total egress HTTP/2 traffic.

Header compression is cutting our overall bandwidth for HTTP/2.

While it does not look like much, it is still more than increasing the compression level for data would give in many cases.

So, better to do this than try to further compress pages.

There's a pretty big disconnect between 0.05% and 1.4%. Believing their numbers would indicate their average page sizes end up being around 7 KiB, or their headers end up being gigantic 4 KiB+ monstrosities, both of which I have trouble believing.

But it's still small. Instead of adding a large chunk of complexity to HTTP/2, how much of that 1.4% do you suppose could be gleaned by removing unnecessary headers like Date, Server, X-Powered-By, cf-ray (on Cloudflare), etc?

I understand that for Cloudflare hosting/proxying others' sites, their options are limited and they have to forward their customers' headers along. But I still think adding all of this is a bad idea. Not that it matters of course, it's already done. The IETF basically rubber stamped SPDY.

I've only glanced at the HTTP/2 spec, but doesn't it allow reuse of headers between requests? e.g. The security headers wouldn't need to be sent with each file, only the first? This could be where the bandwidth savings are coming from.

Yes, it allows re-use.

Access-Control-Allow-Origin is in the static table, and so is Strict-Transport-Security. The rest go into the dynamic table [1], which would be beneficial if they don't change.

Most security headers are hardcodeable, while C-S-P is designed to be a function of the selected resource representation, not some site-wide global. But it's often used in such a way.

[1] https://tools.ietf.org/html/rfc7541#section-2.4

That's what HPACK (which is described in the article) and it's dynamic header table is for. However as soon as you exceed a certain amount of headers (the size which you can store in the dynamic table - typically 4kB) the header table might no longer work as good as possible. That's because the newer headers will always lead to the eviction of older headers in the table, and therefore headers will no longer be cached for the duration of the connection.

There are more gigantic monstrosities than expected.

`content-security-policy` on inbox.google.com is 1.4K and my `cookie` sent to inbox is around 900B. :-(

Security headers are worthy of a different [1], lengthy conversation, but they were borne out of need to override defaults about a browser's security features.

Sometimes, for different features, those defaults are permissive, and sometimes they're restrictive, provided the browser supports that header -- and most of them were designed to be backward-compatible so that their truth tables are complicated.

But realistically, consider C-S-P as metadata about the resource representation about to be delivered in the body, instead of a property of the HTTP response itself. In fact, the naming of the header 'Content-[...]' is consistent with the semantics of RFC 7231 [2] which defines a number of different 'Content-[...]' headers, like 'Content-Type' and 'Content-Language'. You can mentally move C-S-P from the HTTP headers into a meta tag (you can do this actually [3]).

[1] https://hn.algolia.com/?query=niftich%20"damn%20header"&type... [2] https://tools.ietf.org/html/rfc7231#section-3.1 [3] https://w3c.github.io/webappsec-csp/#meta-element

The net effect is much larger than the smallish overall effect appears to be. The request is much smaller and this has a surprisingly large effect because modern consumer connections are asymmetrical as much as 20/1. Those requests take 5-20 times as long per byte to send over the wire. Shrinking them this much decreases total request -> response time a lot more than it looks

You get 192 -> 100 bytes reduction if there's only one response. The compression dictionary(dynamic table) is shared, therefore much more bytes can be saved in the following responses(assuming they contain similar headers).

Many of your responses should be "304 Not Modified", in which case you don't send anything back but the headers. Or, for API calls, you may have a tiny response, or no response other than headers. In all of those cases, going from 192 bytes to (checking) 148 bytes the first time, or 6 bytes for a subsequent request, might make the difference between fitting entirely in the initial response packet or requiring an additional packet.

In addition, hpack also allows the client to compress their request headers, which decreases your incoming traffic. And in that case, the majority of requests will have very little data in them; ideally, only the path will change between requests.

I think the most important gain is that client requests can fit in fewer packets -> fewer RTTs (critical on poor networks) -> lower latency to first paint -> better UX.

Don't forget If-None-Match, If-Modified-Since, Origin, Referer

> Many of your responses should be "304 Not Modified", in which case you don't send anything back but the headers.

A 304 should be able to fit into a single frame, unless you're doing things very wrong. You don't even need headers in that case.

> In addition, hpack also allows the client to compress their request headers, which decreases your incoming traffic.

Yeah, that was the DoS concern I had. Lots of web server software checks the incoming header sizes don't exceed a certain length. But with huffman expansion, you can't quite make an INFLATE bomb, but if the implementation doesn't have a backout function during the decompression, you could consume a huge amount of memory by abusing this.

> ideally, only the path will change between requests.

Can you put the path at the bottom of an HTTP/2 request? Otherwise it would be tricky with lots of bit-shifting to reuse existing compressed headers but insert new paths onto the top of each one.

HPACK on request headers does have a significant impact due to TCP/IP slow start and high latency of mobile networks. 100 requests per page is common (http://httparchive.org/interesting.php), so it quickly adds up, especially for sites that use lots of cookies (well, sites shouldn't use tons of cookies, but in the end it was easier to change HTTP than the sites).

Relatively large request overhead of HTTP/1 caused headaches with icon fonts, sprite maps, and gigantic compiled JS bundles. The aim of HTTP/2 was to make requests so cheap that we could go back to having individual icon files and unbundled ES6 modules.

You know, I think this is my real disconnect with HTTP/2.

To me, I see the problems that make HTTP/1 slow, and so I say, "okay let's address those issues." So I'll inline my includes, use spriting if I have a bunch of icons to display, cache things where possible, perform keep alive where I can, drop the use of giant JS libraries, don't use webfonts, use subdomains to avoid cookie pollution, etc. Doesn't take long.

HTTP/2 seems more like, "how can we let people keep their bad practices the same yet try and speed their stuff up a bit?"

And that's fine and I'm okay with the latter existing to make web sites easier and faster. But I don't like the presumption that HTTP/2 is inherently superior to HTTP/1, which is what 99% of these HTTP/2 explanatory articles promote. Complexity adds a dark cost. I think there's a lot of value in someone being able to write a client that can interact with my site in 200 lines of C code with no library dependencies. I feel like we could have put more effort into our libraries and web packages to make HTTP/1 operate much better instead, and kept the wonderful simplicity of HTTP/1.

While it does make (previous) bad practices perform better (not necessarily a bad thing) it also improves the performance of highly optimized site. For example it enabled far more granular caching. While you might convince people to bundle their assets they aren't going to reduce the amount of content on their site. They will try to optimize it but there is a certain amount of entropy they want to present to the user. If all of this is inlined you will have to request it on every page load. Sure you can be very clever and have a small number of customized bundles where you load a subset for every page, but that is very non-trivial and when we can make that both more trivial and more granular with changes we already needed in the protocol I don't see why we wouldn't do it.

We already needed multiple requests on the same connection because not even the top sites are one request to load (they are very, very few of these). So all of these "bad practice" improvements are essentially "free".

Also I see no reason to try to enforce habits that were formed to work around an insufficient protocol when we are designing that protocols replacement.

> If all of this is inlined you will have to request it on every page load.

True, it's an interesting trade-off. If you inline your CSS and Javascript like I do, then you can't afford to have 100KiB of CSS and 2.4MiB of Javascript code. But you can at least merge all CSS to one file, all JS to one file, and use a combination of max-age and etags to only pay the cost once.

When I look at site waterfalls today ... cnn.com for example is making 20 separate Javascript file requests against its own domain. They don't have to develop their entire JS backend in a single file, but when deploying to production, they should be merging them.

There's also the corollary that trying to put everything on the same connection (both HTTP/2 stream multiplexing and HTTP/1 file merging) tends to have disastrous consequences when you start experiencing 2% packet loss or higher. Great video with charts on that here: https://www.youtube.com/watch?v=0yzJAKknE_k

> We already needed multiple requests on the same connection because not even the top sites are one request to load (they are very, very few of these)

Oh yeah, it's definitely rare. cnn.com hits me with 110 connections to load their homepage.

My site (byuu.org) does it in a single connection, though. Although I'll admit I haven't won any design awards yet for it ;)

The thing is, I wanted to optimize load times, so I learned the best practices and implemented them. If other sites are concerned about HTTP/1 responsiveness, they could easily do the same. I'm actually paying a small bit due to being HTTPS-only with strong 256-bit crypto and compression necessarily disabled. Everything always seems to be a trade-off, eh?

> Also I see no reason to try to enforce habits that were formed to work around an insufficient protocol when we are designing that protocols replacement.

Well again, to me, things should be as simple as possible. People hate C++ because it has ten billion features and you need a large team of engineers to implement even a basic compiler for it. With C, a single developer can do that.

I'm not saying we should've been stuck with HTTP/1 forever, nor with C forever. But certainly we could have started with HTTP/1.2 before turning the HTTP protocol into something that needs a team of engineers to implement in a product from scratch. Server-Push would have been a very simple header to add, for instance.

Yes, the complexity is hidden to the average web developer that installs nginx, flips a switch for HTTP/2, and calls it a day. But as you and I know, that complexity leads to bugs, vulnerabilities, inconsistencies between implementations, lack of choices in software (monocultures), etc. All very bad things. Further, we're stuck with this for at least 20-40 years now. Just like we still have HTTP/1.0 servers. Every new HTTP revision is one more version your software will need to support. How fun do you think it'll be to write an alternative to cURL once we're at HTTP/7? (assuming new revisions pick up the pace as they've suggested it would.)

Http was not designed for what we're using it for. The biggest problem is numerous requests, which is caused by http using TCP underneath, not by multiple requests in themselves being bad.

Bundling is not a best practice because anything about it is inherently better. It's only better because http is bad at delivering multiple responses efficiently.

I think HTTP/2 is a quite reasonable compromise compared to the ideal, which would be scrapping http and starting over. The biggest change is to multiplex requests over a single TCP socket, something that's been done by many other protocols over the years. The added complexity is needed because the alternative, running modified http over UDP, would break so much of our infrastructure that it's implausible.

Does anyone have any practical experience with client certificates and HTTP/2?

https://news.ycombinator.com/item?id=13022596

I had to abandon [client certificates] because of HTTP/2. If one site on the web server uses TLS client auth, and then you go to another site on the same server you receive HTTP 421 Misdirected Request, because of connection reuse. And almost none browser can deal with them correctly (or could not few months ago) - I'm looking at you Chrome, mobile Opera etc... -samsk 2016-Nov-28

Chrome doesn't deal with (client) certificates well regardless of the HTTP version in use, to be honest.

    Response #1:
    date:Wed, 07 Sep 2016 21:41:23 GMT 
    expires:Wed, 07 Sep 2016 21:41:53 GMT

    Response #2:
    date:Wed, 07 Sep 2016 21:41:23 GMT 
    expires:Thu, 07 Sep 2017 21:41:23 GMT

> Although the two expires headers are almost identical, they can only be Huffman compressed, because they can't be matched in full.

It seems like it'd be be an easy win to round expiry times to the nearest minute/hour/day. At least for requests such as these where that value is one year in the future.

This. For HTTP 1.1 clients, Cache-Control already overrides anything set in Expires (a header dating to HTTP 1.0), so losing some precision on an approximate guess far into the future that's bound to be ignored by most clients should be a sensible trade-off.

No offense. But if you had a magic way to reduce HTTP headers to zero, it would have no impact on me as a user.

The fact that a web page loads 40+ domains? Yeah, that's a problem.

The fact that a web page prioritizes serving me an ad so everything blocks until it gets that ad? Yeah, that's a problem.

The fact that a web page downloads 25 Megabytes of data? Yeah, that's a problem.

HTTP headers getting compressed? Not even on my radar.

Your complaints about the web are valid but totally out of scope for the HTTP spec.

Asking why everybody is wasting time on something THAT WILL HAVE NO IMPACT ON END USERS is quite in scope, thanks.

This benefits a very small number of companies who manage to serve very small, highly optimized payloads and nobody else.

And everybody else pays the cost in terms of complexity, security holes, and implementation bugs.

More efficient protocols mean lower costs for those companies. Lower costs potentially means fewer ads to make the same amount of money. Not every technical advance needs to have a direct impact on end users to have a positive indirect effect.

> This benefits a very small number of companies ... And everybody else pays the cost in terms of complexity, security holes, and implementation bugs.

Yes, exactly!! SPDY, QUIC, et al are designed by Google for the Googles and Facebooks of the world. It is not prioritizing what most sites need most. Hell, most sites don't need this period. Optimizing even the most generous 1.4% that Cloudflare claims isn't going to have any meaning to sites ranked below 100,000 in Alexa (eg 99% of them.) They're not going to run out of bandwidth, and their traffic means they're only using ~3% of a $5/mo VPS.

You want software and website monocultures? This is how you get it. Unbelievable complexity creep. All the fun we've had with OpenSSL vulnerabilities? Get ready for them in the new QUIC-HTTP/2 servers.

TLS alone already basically makes it impossible to write your own web server anymore, but at least that's a necessary evil due to ISP and state actor spying abuses. If you're okay with a world where everyone has the choice between nginx or Apache, I guess that's fine. I'm not, though.

You are thinking narrowly about web sites. Lots of other things use HTTP/2 and you will care about performance of those connections (e.g. an API for your favorite app).

the worst thing of the spec is that it writes stuff like:

    An endpoint might choose to close a connection without sending a GOAWAY for misbehaving peers.

But it does not define how a misbehaving peer could look a like. Or

    WINDOW_UPDATE or RST_STREAM frames can be received in this state for a short period after a DATA or HEADERS

Actually it's really really hard to get such a time right. Since it's extremly depending on the client. Also the http/2 state machine is extremly complex vs. the statefulness of a http/1 client (no state machine at all). Also the spec gives a lot of weight to the server/client where they can treat some errors either as a stream only error or as a full error (i.e. goaway, reconnect/disconnect). Not sure if it is good at all.

Push Promises are probably not even useful to the most servers/users. And also have a flaky specification. It's okai to send them between data frames, but it makes no sense to do it.

WebSockets don't work over http/2 and even with a extension they would disconnect every 31-bit/2 streams (or less).

I think that http/2 is a good thing, but at the moment I would rather count on http/2.1 that will get everything correct and made stuff significantly easier.

P.S.: Hpack is probably the easiest to implement/get right

I don't think performing a clean shutdown for misbehaving clients is that important, as they are misbehaving anyway. The only thing were it helps is for debugging the misbehaving application: The developer can see the error message and see something got wrong instead of only seeing a closed connection. In doubt closing the connection is probably always the simplest way to handle invalid situations, as libraries will always have to deal with that situation. Handling close/shutdown/goaway messages is much harder. I also don't consider frames which are received for already reset streams as a big issue. Just ignore them if they are for an unknown stream ID which is lower than the highest established one.

However I agree there are a few rough edges in the HTTP/2 spec. The worst one for me are race conditions around SETTINGS negotiations. If one implementations wants to use a low window size or HPACK buffer size it must still cope with the fact that the remote might send more data before it knows about it - which it can only answer with a stream reset. This in turn might cause many clients to assume that the server is not working or shutting down and to create a new connection and try the same again. The only sensible way for SETTINGS in my opinion is to use the default settings or higher ones - which are unfortunatly so high that it makes HTTP/2 a poor fit for embedded or IoT applications, where it could otherwise shine due to the less demanding binary protocol.

The other thing that I don't like too much are the headers in special header frames without flow control. Getting overall flow control correct (and avoiding that attackers can exploit this) is hard. And from a layering perspective it would have been much clearer if there would have been a lower layer which allows to create flow-controlled multiplexed byte streams and a higher layer which defines on how to transport headers and body data inside these streams. Now we have one big HTTP/2 layer which covers everything. But I guess that was done to get HPACK working. HPACK itself is IMHO ok.

I recommend watching this video by Fast.ly on some real metrics comparing HTTP/2 and HTTP/1:

https://www.youtube.com/watch?v=0yzJAKknE_k

Overall, the performance gain is not as much (if any) compared to the hype we hear.

If it's the video I'm thinking of they ran a huge number of tests, introduced packet loss and saw some issues.

There's a few challenges here…

WebPageTest uses DummyNet to simulate packet loss and in a HTTP/1.x scenario the not all TCP connections in the test will experience packet loss, whereas all the HTTP/2 ones will. Now in the real world if there's packet loss on route how likey is it that TCP connections experience packet loss?

Our H2 implementations are still young, this Cloudflare work is one example, until they made this patch nginx didn't support HPACK and there are other issues in other browsers and servers too.

As the implementations mature and we understand how to make the most of H2 then performance gains will appear - there's already evidence from people like the FT that there are gains to be made now.

Quite.

Real world experience beats synthetic tests. Shipping counts for a lot.

In the time that some other competitors have been talking about HTTP/2 we've had it out for a year and have been experimenting and improving (server push, HPACK, ...). Clearly, HTTP/2 is not the one-size-fits-all-answer-to-everything, but it amuses me to see competitors not shipping, yet talking a good game. See this sort of thing with HTTP/2, IPv6, ...

Every time I look at the hype pages, it's always these unbelievably ridiculous edge cases.

"Look at how much faster HTTP/2 is loading this page that is nothing but an array of 64 x 64 icons that are each 16x16 pixels!" -- well of course that's going to be slow with HTTP/1 if each icon is unique and not cached. But who does that? If that pathological case is yours, then sprite the icons and now it's one request, done.

Performance gain is most likely even negative if you use it for transferring bigger amounts of data (file uploads or downloads) - since the stream multiplexing layer will often lead to an additional copy for each sent/received byte. But that's probably not what the majority of HTTP is used for. On the other hand it will really help all the pages that are making dozens of concurrent small requests, which can't be handled in parallel through HTTP/1 due to connection limitations.

I wonder if HPACK will discourage the creation of new headers... since they won't be in the static dictionary. Instead existing headers will be overloaded (not so pretty).

I doubt that because the information added would be similar in size. Contrast `Foo: old-value unrealted-option=bar` to `Unrelated-Option: bar`. There is almost no difference (only the overhead of another header in the protocol actually. They would have to get way more creative in the representation to get any real gains.

Also the HTTP authors haven't been too worried about header size in the past so I doubt this will change just because we have a new protocol.

Whats with those header fields that start with a colon:

  :authority:blog.cloudflare.com
  :method:GET
  :path:/assets/images/cloudflare-sprite-small.png
  :scheme:https

Isn't that going to confuse every header parser.

That's just a way to spell out the HTTP 2 request-line (its http/1 equivalent would be the very first line of the request like `GET / HTTP/1.0`) in human-readable form.

No parser is ever going to parse these strings.

HTTP2 supporting libraries see the well-known binary values and libraries without support for HTTP2 will fall-back to HTTP1 anyways where the human-readable form of the request-line is, well, just the request line itself.

Thanks, now I know.

They are called pseudo-headers. They are used to transfer the values of the traditional start line of requests and responses. As in HTTP/2 there is no such start line they are encoded like other headers, but are using the colon in front of the key to denote them as pseudo-headers. Each request must contain all the required pseudo headers (:method, :path, :scheme) - same for the response.

And no - this doesn't confuse a header parser since they are not parsed with an HTTP/1 header parser but with the HTTP/2 header tranmission mechanism (HPACK - as shown in the article). Before handing the request to your traditional web app the HTTP/2 server would remove those values from the set of received header fields and instead put them into the request struct.

Significant smaller headers means significant smaller requests. That allows the client to inform the server in a shorter time more comprehensively which set of files are needed. Especially in the mobile use case where bandwidth in the up-link is limited and the channel is not always stable this may really improve user perceived performance.

I don't worry about compression effort too much - deflate has been used for a long time.

I see a lot of patches for nginx coming from cloudflare, how much of their infrastructure is based on nginx, anyone know any good articles on their internal setup?

I don't think we've every done a blog post on the overall architecture of a machine. But as of today it's a mixture of nginx (our own fork since nginx folks won't accept all our patches), rrdns (our own DNS server written in Go), a whole load of Lua (because we use OpenResty), a whole load of other services written in Go, ...

Depending on the protocol (or version) traffic may or may not be served via nginx.

Thank you, would love to see a blog post like that outlining the architecture you have ended up with and how it all fits together.

>nginx (our own fork since nginx folks won't accept all our patches)

Is there any interest in publishing this whole fork? I know you already publish separate modules to github, beyond that is there crossover with nginx plus functionality they don't want to accept?

The problem with publishing it is that we have to maintain it and sometimes we put stuff in there that we wouldn't normally want to make public or it would be a pain to extract because it's linked to our infrastructure.

Open sourcing stuff has a real cost. We do as much as we can.

But as of today it's a mixture of nginx (our own fork since nginx folks won't accept all our patches)

Could you elaborate, within the limits that you're allowed, on the patches that got rejected? I'm curious as to what they were about and why they didn't make it in.

For example, we support both SPDY and HTTP/2 and will do SPDY if an older client can't do HTTP/2. NGINX guys didn't want that.

We tend to publish rejected stuff as patches: https://github.com/cloudflare/sslconfig/tree/master/patches

Awesome, thanks! I can sort of understand not wanting to maintain the SPDY code but I can see how that would be useful to CF to have. I'm a bit disappointed the chacha20 ones aren't making it in.

I'm curious about the Dynamic TLS records patch. Could you or someone else that understands what that's doing elaborate on what's going on?

Great savings. It would be great to know which upstream version of nginx these "full" HPACK patches were merged to. Does anyone know?

I had asked the same question on the blog post, and the author responded there saying the patches were not integrated into vanilla nginx yet, only "upstreamed" to the nginx-devel mailing list in December, 2015.

https://blog.cloudflare.com/hpack-the-silent-killer-feature-...

Reply on news.ycombinator.com