Why I tend not to use content negotiation
htmx.org
htmx.org
This means if you have an endpoint that returns HTML or JSON depending on the requested content type and you try to serve it from behind Cloudflare you risk serving cached JSON to HTML user agents or vice-versa.
I dropped the idea of supporting content negotiation from my Datasette project because of this.
(And because I personally don't like that kind of endpoint - I want to be able to know if a specific URL is going to return JSON or HTML).
Interestingly, one of the big CDN providers did have controls in their UI for explicitly allowing/disallowing Vary header entries but they disabled it for us at some point (e.g. it was still in the UI but greyed out). I assumed once we hit a certain level of traffic it was too computationally expensive? Ever since, I've avoided any kind of fancy header/response variance in APIs just in case I end up in the same situation. It is rarely a necessity. IIRC, the only thing they continued to support variance wise was gzip (e.g. content-encoding).
It's also worth noting they were extremely conservative with query parameters too. Also to reiterate, this was very high traffic and high volume with expectations of low latency, so probably not applicable to most people using CDNs for static website assets.
Seems strange; AFAIK in e.g. Varnish, Vary just means you get more "stuff" tacked onto the buffer that gets built from the request and then hashed to create the cache key.
And actually, come to think of it, if memory for N concurrent in-flight requests is the concern, then you don't even need an actual (dynamically allocated) buffer, either; presuming you're using a streaming hash, you can feed each constituent field directly into the hasher, with only the hasher's (probably stack-allocated) internal static buffer for blockwise hashing required. (Which you're gonna need regardless of whether you're doing any Vary-ing.)
So it's really just a question of how many CPU cycles are being spent hashing. And it's likely just going to be a difference between hashing 300 bytes (base request — hostname, path, headers that are always implicitly Varied upon) and 350 bytes (those things, plus whatever you explicitly Varied) per request. Doesn't seem like too much of a win... (especially when hardware-accelerated hashing ops operate on blocks anyway, such that you only get stepwise cost increases for every e.g. 128 bytes.) I wonder why they bothered?
If this had been a necessity then I would have probably dug into the request more deeply. It was a "pick your battles" kind of thing. Extremely low cost on our side to change, no reason to bother if they claimed it would decrease problems on their side.
I think the best suggestion was in another thread by @johncolanduoni where he pointed out the difficulty of storing, distributing and retrieving the metadata per-resource that would be necessary for each PoP to correctly determine the Vary requirements at request time.
I think that is probably why some CDNs have a single configuration for any HTTP headers you want to vary on (e.g. Cloudfront allows you to specify a global configuration for a distribution that takes into account specific headers). This avoids the problem of both per-resource and inter-datacenter consistency that relying on the Vary header might cause.
> The Vary HTTP response header describes the parts of the request message aside from the method and URL that influenced the content of the response it occurs in.
In other words, if the server backend has a resource with representations that Vary on header values {A,B,C,D}; and one client sends req headers {A,B} — then by the standard they should only be told `Vary: A, B`; while if another client sends req headers {C,D}, then they should only be told `Vary: C, D`. The client should not be told in the Vary response header, about request headers they didn't send.
So it's not just that you can wait for the backend to send a `Vary` response header, and then medium-term cache the value of that header in the cache-policy metadata for the cache key. Instead, on each response, you need to
1. collect any additional Vary fields from the response and add them to your cache-policy Vary set; and
2. have some idea of what the "default header value" would be, to use as a fallback value when computing the cache key, for each header that isn't sent, when it's part of the active Vary set, so that you can dedup requests that explicitly send the header with value X, with request that don't send the header at all but where the default value would be X.
3. Also, ideally, you have a library of normalization transforms for the value of each header used in Vary, to decrease cardinality (the approach of this taking up the majority of the page space on the Varnish docs for Vary: https://varnish-cache.org/docs/3.0/tutorial/vary.html)
And the knowledge required to do all this correctly is really... not knowledge that a middlebox has any good way of acquiring.
This is starting to feel like a design smell in HTTP. Maybe zero-RTT content negotiation is misguided?
What if we instead did content negotiation like this (which — correct me if I'm wrong — would be a mostly ecosystem-backward-compatible change):
- if a resource negotiates, then by default, the server will send a 406 error response for all attempts at retrieving the resource. It sends this because the client itself needs to prove it knows what fields the resource varies on — and, of course, it doesn't know (yet), because nobody's told it yet. This 406 response contains a novel "Should-Vary" response header, informing the client of what it should be sending.
- to actually fetch a resource representation, the client is then expected to make the same request again, but this time, sending an Expect-Vary request header, the value of which matches the Should-Vary header value it saw from the server. Note that unlike with the Vary response header, this Expect-Vary request header should include header names that aren't part of the set of headers it's sending. (And/or, this list should force the client to emit explicit headers with its choice of implicit-default values for any headers listed in its own Expect-Vary header.)
- Upon receiving a request for a resource that negotiates, where the request has the Expect-Vary header set, the server will first verify that the Expect-Vary header value matches the Should-Vary value it would return for the resource, and either matches or is a superset of the Vary value it would compute as the response header given 1. the resource and 2. the rest of the received request. If this verification fails, that's a 406 again, sending Should-Vary again. If the verification passes, and the rest of the HTTP state workflow goes through, then you get a 2XX response. This 2xx response has the old Vary header as part of the response — but it now only exists for ecosystem back-compat.
- If a client thinks it knows the right Expect-Vary header to send, it can try sending it as a request header in the initial request. After all, the worst that can happen is the same 406 error it'd get otherwise. As well, the observed Should-Vary response header value of a resource can be cached basically indefinitely by the browser in its Expect-Vary cache, since the next time it changes for a resource, the browser will try its cached value for Expect-Vary in the request, and get a 406 response that tells it the new Expect-Vary value it should be using instead.
- Optionally, for efficiency, there could be introduced an Others-Should-Vary response header with the value being a path pattern (similar to a Set-Cookie Path field), which specifies other path prefixes for the host that should all be assumed by default to have the same Should-Vary header value as the response does. Potentially, a Should-Vary response header could also be sent in OPTIONS responses, to set a fallback assumed Vary value for the HTTP origin as a whole. (Clients are already requesting OPTIONS for CORS anyway; may as well give them some more useful information while we've got them on the line.)
With this design, middleboxes could safely trust the client's Expect-Vary header and use it to build the cache key — as long as 406 responses aren't cached.
Something for an RFC, maybe?
Some effort to clean up the useless and duplicate features would be good.
If you're talking to an API, you should know what that API can produce.
And if you're talking to a webpage, this isn't an issue.
Well as you're quite interested in APIs: the accept header is a fantastic way to version a REST API, and even has a built in way to handle clients that can talk to multiple versions of the same API.
Your app could request just 'application/vnd.foo.v1+json' from an API and if it's a centralised service that may be fine.
If the API your app talks to is something that's deployed to customers, or rolls out to regions incrementally or whatever, and thus can be at different versions, you need a way to handle that: the Accept header has you covered.
The Accept header does nothing that URLs can not.
URLs are opaque identifiers that only have meaning to the server that generated them. This makes any meaning implicit, where the accept header is an explicit and documented part of the interface. Maybe not much difference in practice for code you've interacted with, but that doesn't mean no difference at all.
If curl has trouble setting headers, that's a curl problem.
Of course you can have a unique URL for each format and avoid the accept header, just like you don't technically need more than GET and POST methods.
What kind of situation would you ever have a request like "Uh I want json the most but XML could work". Either the backend serves it in a format or not, just directly request what you want.
The accept header is passed along, so your server can respond however it wants for dynamic content/not cached by CDN.
E.g.
/fred.html - returns html
/fred.md - returns markdown
/fred.json - returns json
/fred - defaults to html
Though he was a strict fundamentalist restafarian in other ways he held that this had the benefits of
1. Working
2. Being readily understood
I’ve seen worse ideas. Only downside is that the urls look a bit ugly.
https://example.com/#!/
https://example.com/#!/about/
https://example.com/#!/users/
But that was too ugly for URLs, so modern websites now use browser history APIs just so they can remove 3 characters from the URL and do stuff like:
https://example.com/
https://example.com/about/
https://example.com/users/
Sure, the difference is not much, but to me it still seems like a waste when the response could have been cached already like in the first example.
Would back/forward buttons continue to work if we decided not use browser history APIs?
I could try with something like React, but that's too much for a quick test.
.replaceState() will replace current url without putting it in the history, so back won't work, while .pushState() does.
So they invented the history API's so SPA's could stop breaking some of the user expectations they were breaking.
https://developers.google.com/search/blog/2009/10/proposal-f...
If /fred returns HTML, then the URL is perfectly fine, since the only real consumer of fred.md or fred.json are automated systems/API clients, and they couldn't care less what the URL looks like, only that it's predictable.
github.com/[username].png
github.com/[username].keys
github.com/[username].gpg
What about any of that is "secret?"
This endpoint does not work.
Well, at least the appending .json to everything.
Together with that, you can also go for putting it in the `hash` part of the URL, if you want to keep it a secret from the server, as the server doesn't receive those parts, `/fred/#format=md`. In this case, it doesn't make much sense, but useful to know for other things, like keys and whatnot.
Server has no need for it.
Vary: Cookie, Authorization, Accept, Accept-Language
It's a documented decision that they've made: https://developers.cloudflare.com/cache/concepts/cache-contr...
> vary — Cloudflare does not consider vary values in caching decisions. Nevertheless, vary values are respected when Vary for images is configured and when the vary header is vary: accept-encoding.
When building a public scientific database I really want the URL identifying the item in the database to return the page for that item when I enter it in a browser but to return the appropriately structured data for that item when requested with "Accept: application/ld+json" or "application/rdf+xml" by a linked data library.
So it's unfortunate that there's no good way to support this with common CDNs.
Of course I always make it so that appending "?type=json" or "?type=xml" gets you the appropriate document.
That's not true. They do, but you have to add it to the Vary header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Va...
This way user-agents know that the Accept header is involved in the final form of the response. (As another example, Firefox also doesn't take Accept into consideration when caching locally by default, and it does with Varry: Accept)
Their documentation specifically mentions Vary as non-functional[1] outside a paid plan, and even then only for images[2].
[1] https://developers.cloudflare.com/cache/concepts/cache-contr...
[2] https://developers.cloudflare.com/cache/advanced-configurati...
A Vary field containing a list of field names has two purposes:
1. To inform cache recipients that they MUST NOT use this response to satisfy a later request unless the later request has the same values for the listed header fields as the original request (Section 4.1 of [CACHING]) or reuse of the response has been validated by the origin server. In other words, Vary expands the cache key required to match a new request to the stored cache entry.
[...]
When a cache receives a request that can be satisfied by a stored response and that stored response contains a Vary header field (Section 12.5.5 of [HTTP]), the cache MUST NOT use that stored response without revalidation unless all the presented request header fields nominated by that Vary field value match those fields in the original request (i.e., the request that caused the cached response to be stored).
Notice that this is the HTTP "Semantics" RFC, not the "core" one (Message and Syntax) which is the venerated RFC-7230 [1] which is indeed quite simple for such a widely used protocol.
This RFC only defines a handful of "header fields", almost all of which necessary for actually being able to "frame" (find the beginning and end, applying decoding if specified) the HTTP message : https://www.rfc-editor.org/rfc/rfc7230#section-8.1
Yes, I know (and personally experienced) that most clients forget to include "Vary" field, or don't know that it exists, or the problem that this header solves exists) but when they do include it, they actually mean it and rely on it being honoured, one way or another.
PS. By the way, 7230 is obsoleted, you're supposed to use 9112 now.
It seems like building your application around a single API that is also used to provide data externally saves you time, but you end up polluting that API with presentation concerns needed to drive the application's reports/grids/views. It's not worth the mental energy to consider how changes you need to make for presentation might affect the 'purity' of your public API. Returning hypermedia from the 'internal' API just forces that separation: there's no expectation that this 'data' is being returned for consumption by anything except the app that uses it.
Even worse if the json api is public, as it may need to restrict capabilities.
I do tend to prefer actual file extensions though. Friendlier for humans (curl https://endpoint/item.json vs curl -H "accept: application/json" https://endpoint/item), and it is visible in logging infrastructure, sharable, etc.
http://endpoint/item?format=json or ?type=json
I never ever had a problem with that. The only reason to use a file extension would be if the request would take no query parameters.
It would never occur to me to use header information for this.
What I don’t necessarily like about query params for this, is that their usage implies filtering. It’s a cultural assumption but it’s there.
Another point: you might actually have a literal file at a similar path that you serve, typically when you cache responses (managed cache or straight from your app server). Using file extensions gives you a bit more natural affordances. It’s just overal less clunky.
Say, an RSS feed being served as a formatted and styled page to a browser, or a client that accepts the usual XML.
I sometimes prefer file extensions, but then consuming code can often get filled with appending or removing or changing formats.
Broadly, I like content negotiation as a concept because it describes what you actually want and is much more typed (as far as it can be). Adding file extensions feels very stringly typed processing.
Both have problems, you pick your poison and look wistfully at the greener grass on the other side.
Side note: I guess a better title for this blog post would be "Don't share endpoints between machines and humans". If the consumer of an endpoint is a human, it's probably a bad idea to make machines use the same endpoint. Content negotiation is fine, for example if I want to have an API return binary data instead of JSON (if, for example, I'm on an IoT device with sparse resources). This is fine because it's exactly the same API, with just the data format being different. And in both cases it's consumed by a machine/computer.
In the case of the frontend/HTML, the consumer is a human (and not a machine), which — as the article mentions — adds a bunch of constraints that are not applicable to machines: a need for pagination (because scrolling through a thousand results can be annoying), a need for ordering (because I want the most relevant results first), displaying related content (as the article explains). The machine API doesn't necessarily need any of these features.
The author also suggests:
> The alternative is to ... splitting your APIs. This means providing different paths (or sub-domains, or whatever) for your JSON API and your hypermedia (HTML) API.
I believe the alternative has been the norm actually. For example, many front-end frameworks encode UI states in URL, and it's not so sustainable to keep the alignment b/w UI states and data APIs in the long term.
In other words content negotiation is useful to be able to respond intelligibly. If a client asks you for json but not html, it might not make sense to return html.
Lots of use cases could benefit from a hypermedia flow.
I don't think anyone's complaint about server side rendering has ever been that it was too complex.
Then you get into things like how no server side rendering strategy has the equivalent to React “just write an inline helper function for this page”, so you need to create partials in files all over. And what if you’d like to statically verify that your templates have all the content in place? Proper typing? Who hasn’t had a page fall over because there was some missing context variable 3 layers deep.
The thing with the API based flow and client side rendering (with React at least) when you have it set up nicely is “add a list view with pagination and search with a modal to create a resource if needed, and some inline editing” can be done by opening a single file and getting there with 2 dozen lines. Server side rendering strategies in practice tend to buckle a bit if you try to be modular enough, and in many cases you need to open one file per checklist on your project. Code locality issues are real IMO
Disclaimer: I’ve done both, messing around with HTMX on a project and think it’s pretty cool. But it’s mainly because I don’t have all of the nice patterns from work projects that I’m tying this. If startup costs weren’t a thing I’d go with a client side system most days of the week.
But so far we’ve landed on using the django-components library [1] to build something akin to Vue’s single file components. HTML, CSS, JavaScript, and Python “business logic”, all in one file.
That component is then used in the Django templates, or in other component’s HTML, and it all seems to work very React-like in terms of an organized structure of composable components. Combined with HTMX the traditional SSR challenges you mention seem fairly straightforward to handle.
> what if you’d like to statically verify that your templates have all the content in place? Proper typing?
Can you say more about this? Would this be using TypeScript in React to verify types, or something more involved? I’m trying to understand this and what the equivalent might be in this “dynamic SSR” scenario.
I think django-components only has part of the story there, because even if that helps writing out your page, you're still looking at going to the other side of the frontend/backend wall to add, at the very least, an endpoint using that template. And if you're following this whole "hypertext API and data API as separate things", you'll need to also code handling of submitting any data, and routing involved.
I'm not against this stuff entirely, because I think there's a bit of a Go-like niceness to how it's "at worst" boilerplate (that you can abstract over on a per-project basis). But if you have a comfortable, well designed API all that work spread over you templates and controllers might actually all just be in one place.
They added real-time search results to our Django SSR site in 2 lines of HTMX.
It’s the opposite of added complexity.
IMO, it's a great option for rapid prototyping and for projects that can be done with whatever backend stack you already have.
For data, Json is the absolute king. For content, html is king. There is very little to negotiate.
The only case where I needed that feature was when data scientists wanted to download data from my API and needed a bunch of formats (parquet, CSV, TSV). But then they did not really grok content negotiation and asked for a query param. So finally I think this is like a lot of html features: half baked and from a different time. Html would do well to drop it.
Can't we handle this situation by regarding each version of the JSON output as a separate content type (which is arguably the semantically correct thing anyway) and then letting the server pick the more recent output version that the client supports?
I might consider using it for an API whose primary purpose is to support a specific client app that I also control, so users running older versions of the client app still get the desired results, but I don't think the additional elegance is a suitable trade-off for a general-use-by-others API sadly.
Little company-specific binary file formats are just the worst.
I have basically never seen a nice user-facing API when it’s been split out. Sometimes that’s fine, but at least for enterprise use cases having a “real” API just feels like table stakes in so many domains for getting bigger clients onboard.
Originally I thought they meant 2 JSON APIs. One that's tightly coupled with the HTML to handle all the "ajax" requests, and the other for 3rd parties to fetch arbitrary bits of data.
Otherwise, I know what you mean. My company has our internal RPCs and then our customer-facing API and the customer one hasn't been updated in ages and it's just a thin layer over some old internal RPCs we used to have and now we have to maintain backwards compatibility but keep breaking it anyway.
I really think it’s so valuable to take what you use internally seriously, exposing and documenting it to end users is a great way to avoid hackiness, and just leads to more regular designs IMO. More work of course but … not that much in the vast majority of cases
I found a library on GitHub that does exactly this to our app. They reverse-engineered nearly everything. I'm sure we could break the lib if we encrypt a few tokens but no one seems to care. I don't mind either, since we clearly aren't giving users a proper API.
Maybe in the trivial case saying "I'd prefer a JPG to a PNG" can be an assymetrical choice. But in all the interesting use cases I can think of, e.g. where there are competing representation formats, you'd want the server to be able to respond to a HEAD with the choices.
That's the kind of thing you can put in Swagger, but that might lead to hoisting the client's choice into the API, away from Content Negotiation.
The whole point of content negotiation is the client tells the server which types it wants the content in, with weights to determine preference;
The server then works out the best match for what's requested.
Why do you need to ask about what's available? Just ask for it if you support it, and then handle the response based on its type.
As a client, you do know what to ask for: everything you support, listed in order of preference.
Why would you, as a client, care whether the server supports some format that you don't understand?
Hypermedia APIs should be rate limited as well, because otherwise people will just go and screen scrape (like many HN apps do, because HN doesn't offer an API). All a "data" API does is make the scraper's job easier.
> Data APIs typically use some sort of token-based authentication / Hypermedia APIs typically use some sort of session-cookie based authentication
So what. Any web framework worth its salt can support multiple authentication / credential mechanisms - the only "benefit" I can see from limiting cookie authentication is to make life for bad actors with cookie-stealer malware harder (like GitLab does, IIRC).
Then proxies or other IPs will be used anyway.
Announcement https://www.ycombinator.com/blog/hacker-news-api
And "Hypermedia Application Programming Interface" wrong, because generally not for an application at all, but rather a (non-programming, as the author says, "for humans") interface to display multimedia documents ? (I guess you get (inevitable, if not necessarily good) feature creep as soon as you start including something like forms - see also : forms in pdfs ?)
Personally I prefer sticking to the standards. At least that way when you move between projects you know what you’re getting into.
But everyone has their own conventions these days. It’s all fragmenting.
I feel like a good API is limited in what it accepts, and this alone is enough to say one should not do content negotiation unless forced to.
No, I don't think so. Why do you think so?
> I feel like a good API is limited in what it accepts, and this alone is enough to say one should not do content negotiation unless forced to.
Content negotiation does not change what the API accepts. It changes the format of the response.
It's ironic he wrote an article about how the industry uses the term "REST API" incorrectly, because he himself keeps using the term "API" incorrectly. If an "API" is tightly coupled to a single application, it's not an "Application Programming Interface"... it's just a part of your application.
An API is supposed to be an interface on top of which multiple applications may rest. Particularly without a specific frontend in mind - so web, desktop app, mobile app, as a component of other services and so on. Obviously if it serves site-specific HTML snippets, that's not the case. The only reason he advocates this whole thing is because without it HTMX won't work, and in this way I find it quite myopic as a position. But if I was pushing HTMX I'd also be compelled to figure out reasons to make it sound good.
So from that PoV, talking about "Content Negotiation in HTML APIs" loses meaning, as what he has is not an API in the first place, it's just his HTML website, but with some partial requests in the mix. And of course you wouldn't mix your API and your site. But this does not imply you can't and shouldn't use Content Negotiation either on your site, or in your API. You simply shouldn't use them to mix two things that never made sense to mix.
A lot of his blog posts would become completely unnecessary if he just says "don't mix your website and your API, and the HTMX partial requests are part of your website, not your API". Alas he's stuck on this odd formulation of "hypermedia API" and constantly having to clarify himself and making things as clear as mud.
Specifically a hypermedia API is something browsers (and the implicit backbone of the internet) understand very well. In fact you have to go out of your way to serve an application that your browser doesn’t inherently understand.
The clarity that seems to be lacking here is not necessarily a fault of the author. We re-purposed some of these terms (REST, API etc.) to serve specific needs. But then kind of lost the understanding of what we had before.
I think that’s not our fault though. Standardization didn’t move fast enough and the quality we needed in the mobile context wasn’t there.
Quoting Roy Fielding:
"The design of the Web had to shift from the development of a reference protocol library to the development of a network-based API, extending the desired semantics of the Web across multiple platforms and implementations. A network-based API is an on-the-wire syntax, with defined semantics, for application interactions. A network-based API does not place any restrictions on the application code aside from the need to read/write to the network, but does place restrictions on the set of semantics that can be effectively communicated across the interface."
Nah I think he's right and it's coherent to avoid HTTP subtleties in that web architecture.