New HTTP standards for caching on the modern web

Digest should just die. It’s a terrible standard that makes it essential impossible to store passwords securely on the server.

> Those headers use comma both to separate challenges and to separate > parameters within a challenge, which just seems mean-spirited.

Indeed. And while the resulting semantic meaning is unambiguous, because both lists allow empty elements (well, you shouldn't emit them but as you note, section 7 says you still need to be able to parse them), it can be ambiguous for the parser whether an empty element belongs to the inner list or the outer list. So you'll likely run in to trouble if trying to use a parser-generator or generic algorithms, even though the ambiguity is totally inconsequential.

In "Scheme1 realm=foo, , Scheme2 realm=foo", is the empty element an empty param for the Scheme1 challenge, or an empty challenge between the Scheme1 and Scheme2 challenges? Answer: It doesn't matter, but good luck telling your tooling that.

> Other things about HTTP authentication that seem mean, dumb, > annoying, or all of the above: both the RFC 2069 example response > and the RFC 7616 SHA-512-256 example response are calculated > incorrectly;

RFC 2069 (rev 1): Eh, the mistake is noted in the errata.

RFC 2617 (rev 2): Gets it right.

RFC 7616 (rev 3): Indeed. This has been reported as errata (way back in 2016), but is not "verified". The report claims that the incorrect values are obtained by naively truncating SHA-512 to 256 bits, rather than making the other changes (using a different H⁽⁰⁾ value) for it to be true SHA-512/256; but I haven't checked this to comment on it.

> RFC 7616's userhash field seems to require the server to do > O(users_in_database) hashes to know what user to operate on;

I'd assume that you'd just store the H(concat(username, ":", realm)) as a column in the database, and select on that. I guess you're still doing O(users_in_database) hashes, but you'd do a hash when you create each user, not re-doing all of them for each request.

> RFC 7235's challenge grammar describes a token68 syntax that really > only is used for the credentials in basic, never a challenge;

Yeah. That's weird.

> RFC 7616 drops backwards compatibility for RFC 2069 even though I > bought a product this year that still uses RFC 2069-style > calculations;

You don't mention that RFC 2617 set up a migration path, which RFC 7616 finally "turned off".

1997 (RFC 2069): Here's a formula.

1999 (RFC 2617): Here's a new more secure formula, servers "SHOULD" set the "qop" parameter to opt-in to it, and clients "SHOULD" respond in kind. But if qop is unset, then fall back to the old formula.

2015 (RFC 7616): The old formula is gone, you "MUST" set the qop parameter to select the new formula.

That was a 16 year migration period where both were supported, during which everyone "SHOULD" have switched to the new way.

> and it's based on old standards that followed "be conservative in > what you do, be liberal in what you accept from others" so RFC 7230 > section 7 has separate grammars for what lists you must send and > what lists you must accept, which further complicates parsing the > nested lists.

You make it sound like they totally changed it; the new list syntax is a subset of the old syntax, not something different.

Eh, there are already lots of places in the spec where it says "don't send X... but if you receive X, you should accept+handle it anyway". This is just one more.

HTTP/4 should send all headers in JSON format and dates in ISO format.

The old RFC822 header format looks so simple, but parsing it is the devil, especially email headers such as "to".

Cache-Control: max-age=0, must-revalidate

Sounds like what you want (presuming your server handles 304 logic correctly)

This is what ETags are for. Upon a user's first visit the server should return an ETag uniquely representing the current version of the page. The browser will cache both the page and the tag. Upon subsequent page visits the browser will send an If-None-Match header containing the tag for the version of the page it has cached. The server should compare the incoming tag with the tag for the current version and return a "304 Not Modified" response if the tags match or a full response with the newer tag in the ETag header if they don't.

It's a combination of different headers that's hard to sum up in a short comment. A good article on the subject should talk about all these headers: Expires, Cache-control, Etag, Pragma, Vary, Last-Modified

Key CDN has an article on it. They certainly would have experience and expertise there. I didn't read the whole thing, but it seems to have it covered: https://www.keycdn.com/blog/http-cache-headers

There's also some interesting exceptions where rules aren't followed. Like browsers typically have a completely separate cache for favicons. I suppose because they use the icons in funny/different ways, like bookmarks.

There are also sometimes proxies (especially corporate MITM ones) that don't follow the rules. Hence the popularity of cache-busting parameters like you described.

as of my understanding of the original design of HTTP, each HTTP resource may state how long itself can be cached in the response header; and the client (browser, proxy, etc) does not have to re-request the resource before the expiry. this is the sandard, so you can not hint that a resource has to be revalidated - in standard way. obviously since then, several tricks emerged, like your mentioned timestamped URL approach - however i'm not sure upto what extent is it standardized in clients to understand that "/path?query" is somehow related to "/path", because originally the request string (path and url parameters) was opaque to the http client, so they should be cached independently. things obviously changed since then. the method i use is to fire a request to the URL which has to be refreshed by Ajax (XHR) with Cache-Control header (yes, it is a request header too), then display the response content or redirect to it.

There's no standard way for one page to invalidate another. I've seen some private patches to do it in squid, but that doesn't help because you want to do it for browsers.

Your options are probably:

a) redirect to a different URL as you've done by appending stuff to it

b) require revalidation on each request, recipies shown by other posters

c) POST to the url you want refreshed; post isn't cachable. Note that you can't redirect to POST somewhere else, but you can do it with javascript.

d) use XHR to force a request as another poster mentioned.

I have this same issue and have been working around it by appending to the URL. I'd like to believe there's a better way, but I don't know what it is. Alternatively, I could just disable caching but that would defeat the point.

On Macs, in Safari, cmd-opt-R is “reload page from origin”, which I believe ignores cache

There's no easy way. One way is to use `max-age=0, must-revalidate` but then your origin server should be optimized for conditional GET requests.

It's a very tricky balance between origin server load and consistency. By deciding to use HTTP cache you agree to eventual consistency and this decision comes with its upsides and downsides.

There has been a proposal in 2007 for a thing called cache channels. It defined a mechanism for an origin server to expose a feed which caches would poll at an interval. The feed would list resources that have gone stale since the last query. This mechanism in conjunction with conditional GET requests would've solved part of the issue of hinting browsers to invalidate their local resources.

Here modern is not description of the RFC iteself,

modern web = today's web

https://www.etymonline.com/search?q=modern

> In history, in the broadest sense, opposed to ancient and medieval, but often in more limited use. In Shakespeare, often with a sense of "every-day, ordinary, commonplace." Meaning "not antiquated or obsolete, in harmony with present ways" is by 1808.

The architecture of the web (REST), always had intermediaries as first-class citizens. I can recommend reading into Fielding's dissertation!

The declarative and stateless message design is partly motivated by the need for intermediaries processing and understanding the messages (ie POST must not be cached f.e.).

I think the introduction explained it very well: The original design generalized too much for the needs of 2021 by putting all caches into a single bucket.

Client<>Proxy hop has very different characteristics than Proxy<>Server hop, so you may want to use different caching strategies for each.

For example, cache on clients for a long time, because the network is slow and expensive there, but cache on the proxy server only for a short time, so that it updates from the server relatively often.

Or you may want a reverse: cache on clients only for a few seconds, because you can't purge clients' caches. But tell the proxy to cache forever, because you will manually purge its cache when something changes.

This was previously sort-of possible with max-age vs s-maxage, but other cache directives don't have two versions. Notably, stale-while-revalidate didn't work.

Stale-while-revalidate has different implications on each hop. On the client it means it's stale until the next page load, which may be undesirable. On a proxy that keeps getting hits all the time, it's only a slight propagation delay.

A lean website (they call out haveibeenpwned) can still get billions of hits. CDNs were initially conceived to help with international / intercontinental latency, which you can't fix by having a leaner website. And there's a heap of problems to solve that you can either spend your own valuable time on, or outsource to a specialist.

Leaner websites can only reduce bandwidth strain, there's no impact on latency, right?

It sounds like you're describing Signed HTTP Exchanges (SXG).

https://developers.google.com/web/updates/2018/11/signed-exc...