IETF: The HTTP Query Method ( Draft)
ietf.org
ietf.org
This feels like a bad idea, since (a) different caches will support different content types and normalize them in different ways, leading to unexpected changes in behavior and (b) some servers may behave differently depending on something that a cache considers to be a "semantically insignificant" distinction. I'm not sure, in other words, if I trust caches to get this right.
It seems like it might be better to require clients to submit requests in a "pre-canonicalized" form, or to have caches allow this behavior but disable it by default.
The query and response MUST be transmitted and stored 'as is' (a sequence of octets).
The query and response SHOULD be encoded in either UTF-8 or WTF-8 https://simonsapin.github.io/wtf-8/
Future standards or non-standard systems MAY use different encodings; conformant implementations MUST NOT alter the sequence of bytes. They MAY perform a validation check and add additional headers.
> WTF-8 must not be used to represent text in a file format or for transmission over the Internet.
Also, as per the rest of my ideal solution, the true encoding is: Sequence of Octets / Bytes
Odd proxy bugs/client quirks with zero/non-zero Content-Length headers in GET requests will remain with us forever
Why would you put a body in a HEAD, OPTIONS, TRACE, or DELETE request (or GET, which will continue in use alongside QUERY)?
And, no, a lot of response also don't have bodies, and that will continue.
The possible creation of extra HTTP resources (response resorces?) seems to me contrary to idempotency. That seems more like the territory of POST.
If two identical QUERY requests might produce different response resources, how to square that with the the fact that QUERY will be cacheable?
> The cache key for a query (see Section 2 of [HTTP-CACHING]) MUST incorporate the request content.
If I interpret it correctly - adding more resources isn't changing the server state but adding more ways of getting to the state.
A way to square creation of response resources with idempotency could be: the second identical QUERY that arrives should always reuse the result resource created by the first QUERY.
If I QUERY the current price of a stock, and then someone else sends an identical QUERY ten seconds later, they might get a different result. This is not because QUERY isn't idempotent.
rfc2616 says:
> Methods can also have the property of "idempotence" in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request.
https://datatracker.ietf.org/doc/html/rfc2616#page-51
Perhaps changes in the underlying data could be considered "expiration issues". Otherwise not even GET could be considered idempotent in many cases.
That's why it's called idempotent - 'doing the same' - rather than impotent.
If a request changes the state of the server and another identical request changes the state of the server in a different way, it's not idempotent.
If a request doesn't change the state of the server at all it is idempotent, even if subsequent requests might get different responses (e.g. the stock quote example in my previous post).
If a request changes the state of the server but repeated identical requests don't have any different effect it is also idempotent. For example, DELETE is idempotent because DELETE-ing something N times is the same as deleting it one time.
“Obsoleted by: 7230, 7231, 7232, 7233, 7234, 7235”
This is incorrect. DELETE is idempotent but changes the state of the server.
No, it's not. That's closer to “safe” than “idempotent” (safe also implies idempotent, but not the other way around), but even then it is not quite right, because even safe methods are allowed to have side effects, but their is guidance about the kind and impact of side effects that it shouldn't have.
If two repetitions of a QUERY request create the same extra HTTP resource(s), then it can be idempotent.
Idempotent means you can't tell the difference between 1 or N requests, not that you can't tell the difference between 0 and 1. Think about PUT, which is also idempotent.
GET can have side effects, and has no difference first and subsequent invocations (because it is safe as well as idempotent). Were it idempotent but not safe, it could have side effects that the client was accountable for the first request, but no different ones of that kind for subsequent uses.
Or maybe we just need distributed garbage collection for URLs.
Printing a URL on paper leaks it. Writing a URL down, or memorising it, also leaks it – but the computer has no way of knowing this has happened.
I don't think this is feasible.
That means that a QUERY request can change the state of the server, for example by creating new resources; there's exactly one resource it's not allowed to change.
If I've read it right.
Idempotent etc in the HTTP specs has always been more or less an attempt at a promise to the client "you should be able to repeat this request if you're not sure about success/failure without anyone claiming to implement HTTP being able to throw the book at you".
Just like GETs shouldn't have side effects. But in practice of course, things like https://thedailywtf.com/articles/the_spider_of_doom happen
[Edit: the return redirect to URL that somehow encodes the query usage is even given as an example in section 4.2]
A GET request might create additional (or modify existing) resources, say if the API exposed it's own log via HTTP.
Both safe and idempotent are less expensive than one might naively think in the HTTP spec (which is good, because the naive understanding, while aesthetically seductive, isn't very practical at all.) Some quotes from the relevant bits of RFC 7231:
“This definition of safe methods does not prevent an implementation from including behavior that is potentially harmful, that is not entirely read-only, or that causes side effects while invoking a safe method. What is important, however, is that the client did not request that additional behavior and cannot be held accountable for it.”
“The purpose of distinguishing between safe and unsafe methods is to allow automated retrieval processes (spiders) and cache performance optimization (pre-fetching) to work without fear of causing harm. In addition, it allows a user agent to apply appropriate constraints on the automated use of unsafe methods when processing potentially untrusted content.”
“Like the definition of safe, the idempotent property only applies to what has been requested by the user; a server is free to log each request separately, retain a revision control history, or implement other non-idempotent side effects for each idempotent request.”
“Idempotent methods are distinguished because the request can be repeated automatically if a communication failure occurs before the client is able to read the server's response.”
GET /api/query
Host: example.org
Content-Type: application/json
{
"a": "valueWith$pecialChars",
"b": "valueWith$pecialChars",
"limit": 100
}
If QUERY was around I'm sure I could've made a stronger case to fix it sooner.
Elasticsearch also encourages GET with body. But a request payload is undefined, according to the RFC:
A payload within a GET request message has no defined semantics;
sending a payload body on a GET request might cause some existing
implementations to reject the request.I think I'm on the side of the PM with this one on both counts. You sound like someone who really cares about efficiency, performance, and edge cases -- a proper engineer. But PMs are supposed to bring us down to earth and say that simplicity and maintainability are more important than saving bytes and to not waste time fixing things that aren't broken.
In a past life I spent so much effort optimizing our stack to lower our AWS bill until a PM sat me down with the company's finances and showed me the teeeeny little bar that was our cloud expenses and then legitimately 20x taller bar that was salaries and basically said that spending money to buy back my or my team's time was more important.
Before or after Content-Encoding: gzip?
Note that using the "X-" prefix has been deprecated since 2012: https://datatracker.ietf.org/doc/html/rfc6648
Creators of new parameters to be used in the context of application
protocols:
1. SHOULD assume that all parameters they create might become
standardized, public, commonly deployed, or usable across
multiple implementations.
2. SHOULD employ meaningful parameter names that they have reason to
believe are currently unused.
3. SHOULD NOT prefix their parameter names with "X-" or similar
constructs.
Note: If the relevant parameter name space has conventions about
associating parameter names with those who create them, a parameter
name could incorporate the organization's name or primary domain name
(see Appendix B for examples).Is this really worth a change to every HTTP client library out there to support this? The limited applications that really need this can easily use POST and document their own semantics around this.
If anything the trend with GraphQL is to ignore HTTP verbs outright because they are limited and inexpressive beyond simple CRUD tasks.
A payload within a GET request message has no defined semantics;
sending a payload body on a GET request might cause some existing
implementations to reject the request.But presumably no-one is brave/foolhardy enough actually to redefine GET as having a semantic body because a bazillion different implementations (clients, servers and middle boxes) probably become non-compliant.
So what actually?
apps that didnt use GET Body, will not care anyway
apps that will use HTTP GET Body will be checked anyway
So, unless somebody downgrades HTTP Server then what could be the problem?
Granted they need to get support for QUERY too but at least it is more explicit then.
An official readonly flag to POST would have been more backwards compatible...
What if we allowed HTTP GET Body?
What's the difference between PUT and allowing DELETE with a body?
The samples should include some non-SQL, completely made up ones, as I think a lot of people are going to fixate on the SQL-like syntax and its associated problems.
Because, like it or not, you're often still sending/receiving documents? :)
Also the semantics of the actions map really well to the use cases still.
You're, of course, more than welcome to avoid the whole thing and use WebSockets or WebRTC data channels with a custom protocol.
Then we'll be ready for INQUIRE too. Which is similar but involves more formalised header definitions.
Following this. We'll get an RFC for OBTAIN, which will be similar to GET but will formally renounce the use of a payload.
Also because a different method means OPTIONS tells you information about what is supported, while overloading GET would not.
And because “same guarantees” doesn't mean “means the same thing”; PUT and DELETE have the same guarantees (idempotent but not safe), but we don't use PUT with no body for DELETE.
Query is a synonym for ask. Request is a synonym for ask.
We are now making request requests?
This is redundant and is not needed.
We need to stop modifying basic protocols like HTTP. We should've stop at 1.1.
The examples are realistic and useful. E.g., Clickhouse uses POST methods for queries, and a ridiculous `&readonly=2` parameter to differentiate modifying queries from readonly SELECT queries.
Is that really what you want from a query operation? I read 'idempotent' as implying that result sets don't change over time, which would be surprising behavior for queries for most database-like things.
It's probably also worth mentioning that SQL's SELECT isn't idempotent in the way HTTP means it, because of the existence of session state, pessimistic locking, and the requirements of higher isolation levels. It would be useful for an RFC to define 'idempotent' in a way that clearly addressed these issues (and, for that matter, the larger topic of sessions/transactions) more clearly.
> When doing so, caches SHOULD first normalize request content to remove semantically insignificant differences, thereby improving cache efficiency
Unfortunately, again when you look at SQL by comparison, queries are not purely expressions of what to return. Practically, they also encode how to compute the query (either explicitly through hints, or implicitly through things like join order). These behaviors are weird, tricky, and change version-to-version.
> The QUERY method is subject to the same general security considerations as all HTTP methods as described in
As another commenter said, this is quite incomplete. Query parameter injection, DoS by locking, DoS by exploiting work the database needs to do to ensure isolation, DoS by extremely expensive query, etc.
> 4.2. Simple QUERY with indirect response (303 See Other)
At least the examples here are naive - most applications don't want query result sets to be easily accessible to others. The semantics of authn and authz need to be really crisp here to make sure that attackers can't access the location of other queries result sets purely by guessing.
At least a "SHOULD use auth" or "SHOULD have large, unguessable, names" would be valuable here.
That's not what idempotent means in HTTP.
> As another commenter said, this is quite incomplete. Query parameter injection, DoS by locking, DoS by exploiting work the database needs to do to ensure isolation, DoS by extremely expensive query, etc.
Is application-dependent and applies to all other HTTP methods too.
> A sequence is idempotent if a single execution of the entire sequence always yields a result that is not changed by reexecution of all, or part, of that sequence.
Which isn't, because of isolation, true in general of database queries. Obviously this is in context of RFC2616 saying that sequences of idempotent HTTP operations may not be idempotent in themselves, but that definition seems very incomplete in the context of database queries.
> Is application-dependent and applies to all other HTTP methods too.
Sure. But I don't think that's a good argument in the modern world. Over the 22 years, we've learned a lot about the security concerns of running secure systems, and it seems reasonable to include those concerns in a section labelled "security considerations". SQL injection is a classic security bug, and should be a key concern of any reasonable new standard for sending queries between systems.
A full security section should probably also mention cache timing side-channels, locking-related covert channels, and other similar concerns that come up when you increase the semantic power of HTTP. It's not that POST doesn't have these concerns, it's that we've learned in the last two decades that they are real problems for many kinds of real systems.
RFC7231:
> A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request.
note the on the server.
(or old specs, 2616: Methods can also have the property of "idempotence" in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request. - again, side-effects, not responses)
How does a read-only database query being repeated cause a change in the database?
https://developer.mozilla.org/en-US/docs/Glossary/Idempotent
b) Some developers are ignoring HTTP idempotency. Not IETF's fault for them abusing GET for deletion.
A command to add a user to the set of users that have upvoted a post would be idempotent. Because you can run it 20x and only the first call affects anything. A command to increase the upvote count for a comment by +1 would not be idempotent.
Not necessarily - plenty of systems offer no such guarantees, and the Web is by design eventually consistent [0]. This is what content expiration and various other cache control mechanisms are for - it's not always so important to get the latest version of a document. For example, the HN logo or index.html can probably be safely cached for days, since they are very unlikely to change, and even if they do, it's unlikely to have a major problem if someone only sees the new version after a few days.
[0] Note that, at the extreme, due to special relativity, there is no absolute notion of "latest version" on the scale of geographically distributed computers: it's physically impossible to say if a request made in China to a server in the USA happened before or after a change on the server, if they happened close enough together - order of tens of milliseconds, an eternity in compute time.
The specs do not talk much about the document changing behind your back, only about the changes you cause by yourself.
1) is it safe to automatically retry the request? - this meshes well with what you're saying
2) is it safe to return a cached version of the response, instead of sending the request again? Idempotence in your sense is necessary but not sufficient for this case - hence the various content expiration and If-Newer-Then etc headers.
Idempotent means the request itself doesn't change stuff, it doesn't mean something else won't.
However, because of the existence of isolation and locking concerns in databases, even fairly simple queries are not idempotent. RFC2616 goes to some effort to (fuzzily, unfortunately) talk about sequences of operations, which would be useful here.
In SQL this is a query:
SELECT a, b, c FROM foo LIMIT 1
INSERT INTO foo VALUES (1, 2, 3)
# An UPDATE to the database
client.execute('INSERT INTO iris SELECT * FROM another_iris_table')
# A harmless "query"
result = client.execute('SELECT COUNT(*) FROM iris')
print(result)
ClickHouse also supports GET as a verb. In addition to URL length issues the query needs to be URL-encoded which makes it difficult to read and debug.
p.s., It's interesting to see my post downvoted. It's more productive to show why it's wrong. I've worked on DBMS connectivity for over 30 years.
See https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-meth...
> The non-normative examples in this section make use of a simple, hypothetical plain-text based query syntax based on SQL with results returned as comma-separated values. This is done for illustration purposes only. Implementations are free to use any format they wish on both the request and response.