URL query parameters and how laxness creates de facto requirements on the web | Hacker News Reader

URL query parameters and how laxness creates de facto requirements on the web

utcc.utoronto.ca

101 pointsoftenwrong73 comments

Years ago I worked on Google Maps, we spent an inordinate amount of time making sure that we didn't break backwards compatibility. Sometimes we would be trying to refactor some old code, but we'd get stuck trying to support some ancient client that was sending us like 4 requests a day with some wacky query parameters.

On the other hand, our Maps still worked on all 5 Blackberries that people were still using.

Were there different user agents or did all clients have to use the same interface?

Err, from a user perspective you actually spent an inordinate amount of time changing things for the sake of it.

It was really annoying.

Google APIs have always been some of the worst.

It honestly felt like ever time I had to use the maps API for a client, you'd changed the whole thing.

Yeah, I think people felt less compunction changing the published APIs. I was talking about the "internal" APIs, especially the tile API which still serves ancient clients.

That is an extremely strange order of priorities, right?

Probably because the "internal" API was used by someone who paid a lot of dollars?

Just in case anyone's curious: there's still at least one BlackBerry user. I'm surprised AT&T haven't shut off the BIS gateway yet, but at least HN loads on this thing.

Edit: Thank you, though, for your effort. I'm not sure if it still works, but I did use the Google Maps client for some time, loaded one .cod file at a time...

-- written from my Bold 9900

Relevant: https://www.tnhh.net/posts/google-maps-insane-backward-compa...

My favorite story to tell in this area: many years ago in the early days of Chrome, there was some logic that followed HTTP redirects that had a cap on how many redirects to follow before giving up. (The rationale here is that if you hit too many in a row you probably had encountered a broken site , like one where /foo?q=1 redirects to /foo?q=2 redirects to q=3 and so on).

As I recall, the cap on redirects was 10, and then Darin (who had previously worked on Firefox) was like "no, you have to allow up to 30 or you break the New York Times". And so, it was upped to to 30. Intuitively I'd think a site that redirected you 10 times was broken, but I guess someone built one that needed more.

PS: Looking online now to confirm my facts, I see one claim that both Chrome and FF limit redirects to 20. So either my memory exaggerated it as 30 or they both managed to lower the limit at some point.

I kind of wish the cap had remained at 10, and the redirection on the NYT was left as a problem for them to fix...

Upstart browsers are not in the position to tell websites what to do; they will just throw up a "your browser is not supported" banner and forget about you. That was where Chrome was in its early days (you were supposed to be using IE 6), and that's where Firefox is now.

We never let economics solve this problem, because nobody pays for web browsers with money. If there was a $10 "sorry, The New York Times violates standards so we don't support it" and a $100 "we'll work around any bug large or small to make it work for our customer" browser sites would be spending money to let the $10-browser-owners see their ads. But, every browser is run as a charity, so nobody gets to pay the actual costs of the workarounds (except maybe in terms of more security vulnerabilities, higher RAM usage, etc.)

I'm not saying this is good or bad, it's just how it is. Browsers are free and will be blacklisted by sites that don't like how they behave. So the incentive for browsers is to do what the sites want, rather than to strictly conform to standards. Shrug.

What could they be doing with all those redirects?

Run you through all the spying services' cookies & other fingerprinters. Just like Twitter's t.co.

IIRC you can mitigate this by lowering network.http.redirection-limit

Wouldn't that prevent reaching the page?

It would prevent reaching the page, and still give your contact to trackers - you might just prevent one of the last ones from running.

The first two are often:

    nytimes.com > https://nytimes.com
    https://nytimes.com > https://www.nytimes.com

And from there, who knows. Some websites append a trailing slash; sometimes they removed it. Some websites send you to /index.php or /home.aspx rather than sitting in the root.

Subpages may use a redirect to update a slug if there's been a title change. Or maybe the website just moved to a new format altogether and is updating old links.

I have seen examples of 4 or 5 legitimate redirects to send users to the correct place. I'm having trouble coming up with 10 possible reasons, though.

Ha, that must be for academic articles as well. They all (well, almost all) have an ID that you can look up at doi.org, which is supposedly kept up-to-date and links to the article at the publisher's website, wherever that is. But then in practice, that publisher's website adds another bunch of redirects because why not.

I had an edge case with an authentication server (think okta) wrapped around a school login system. In occasional cases, with certain clients of the server doing a couple redirects of their own, we'd hit that 20 cap. It's not any individual system being irresponsible, just reusing other systems as they're meant to be. It's kind of like saying having a call stack depth of 50 in software is never acceptable.

> One of the ways that DWiki (the code behind Wandering Thoughts) is unusual is that it strictly validates the query parameters it receives on URLs, including on HTTP GET requests for ordinary pages. If a HTTP request has unexpected and unsupported query parameters, such a GET request will normally fail.

This goes counter to the robustness principle: Be conservative in what you do, be liberal in what you accept from others (https://en.wikipedia.org/wiki/Robustness_principle).

Which we’ve increasingly moved away from because of some of the absurdities (and security problems) that it’s caused.

Nowadays the w3c and es committees work hard to try and ensure no ambiguity is left in the spec, and have tests available early on, and do interop comparisons between engines to see if there’s anything missed.

A big issue with being accepting is that you risk breaking things tomorrow when you no longer want to be as accepting. And it can prevent adding features that would play badly with others' out of spec implementations.

Like if twitter adds these s=NN parameters, that means you can't use an s=NN parameter for your own purposes.

All the worse for the ‘robustness’ principle then.

https://tools.ietf.org/id/draft-iab-protocol-maintenance-04....

(Shame this draft just expired before being finalised.)

The robustness principle is widely considered to be a bad practice these days.

Strictly failing is a good thing, generally. Accepting arbitrary query parameters is pretty much a de facto exception though

You have to ensure you don't over-engineer specifications or it will be ignored as well. HTTP requests are one thing, imagine if we were strict on HTML validation. We now have robust browsers and I am not sure if that is only a bad thing. Although it would be nice if people would check their documents once or twice.

edit: I don't think considering the principle as bad is a sensible approach as it doesn't mean to ignore errors in theory.

> imagine if we were strict on HTML validation

You don't have to imagine it, this actually happened. I don't remember the details, but back in the HTML 4 era, there was XHTML, and if you gave it an XML media type, many browsers would parse it as XML and reject the entire document if there was any error (and display nothing at all but an error message).

There were a number of articles and blog posts about this back in the day.

I feel that you’re right - it’s increasingly considered bad practice - but I’ve not seen this conversation play out.

Of course, this often comes up in collaborative coding exercises (work, open source), so seeing a discussion on the topic would be helpful.

Anyone have a link to such a discussion?

https://tools.ietf.org/html/draft-iab-protocol-maintenance-0...

As an example, rejecting unrecognised query parameters can cause ossification of web protocols. For example, I do a lot of work with OAuth and it is accepted security best practice now to use PKCE [1] on all authorisation requests [2]. However, despite the OAuth spec saying that authorisation servers MUST ignore unrecognised parameters [3], there are many major services that fail if you add a PKCE code_challenge parameter. This means generic client libraries either don't implement PKCE or have to maintain a list of sites that don't work (which gets stale over time) or have some risky downgrade logic.

So there is a tradeoff between strict validation and allowing future expansion and upgrades. By all means strictly validate the parameters you need, but maybe don't reject parameters you don't recognise.

[1]: https://oauth.net/2/pkce/ [2]: https://tools.ietf.org/html/draft-ietf-oauth-security-topics... [3]: https://tools.ietf.org/html/rfc6749#page-19

The same thing happened with TLS!

TLS 1.2 has a field for the protocol version during negotiation. The spec says that you're supposed to proceed with the lower version if the other party claims a version higher than yours.

Turns out in practice that some implementations just decided to give up instead. This meant that it wasn't possible to increase the version number for TLS 1.3, because advertising it would break a lot of people's connectivity. Instead, TLS 1.3 uses the 1.2 version, and added ANOTHER field containing the real version.

So Google introduced GREASE in RFC 8701. Long story short, Chrome now advertises a bunch of bogus extension values, thus making sure that implementations can't simply ignore unknown values.

This makes it a lot more likely that future cipher additions will not break older clients, which is pretty damn important for something as critical as TLS.

I often wonder why implementors do such hacky gymnastics instead of just breaking the clients of the corps that use janky noncompliant middleboxes and causing people to replace their broken shit.

Other than those corps who bought crap, nobody would blame Google, for example, and I bet those orgs would only do so temporarily.

Make life slightly more difficult for those whose poor decisions CAUSED the ossification.

"Last Change Broke It" is the rule. Nobody cares whose "fault" it notionally is that something doesn't work now, they will focus on whatever change made it stop working. This is not least the case when you paid $1M for this security appliance two years ago, and then a free Chrome update broke everything on Tuesday morning.

Are the vendors going to give back your $1M? No they are not. Are they going to fix the appliance? Maybe. Although perhaps not unless they can see how that'll bring in more money. So you blame Chrome, because that's what broke.

Now, for the sake of security sometimes, very gently, the browsers will push to fix things because there's no alternative. They have to co-ordinate (if one of the big browsers defects then users switch to "the one that works" and you all lose) and it takes a long time to do. So that's the last option.

For example, TLS 1.3 as shipped includes an anti-downgrade mechanism. It was not possible to ship this in draft versions so it was not widely tested. It turns out that some middleboxes from popular vendors did something so stupid and dangerous that it broke the anti-downgrade. (Specifically they copied the Server.random field from a session they were having with an actual HTTPS server into the Server.random field they delivered to a browser, even though the standard is emphatic that you need to choose your own random numbers or it isn't actually secure).

So, for about a year Chrome did not enable anti-downgrade in order to allow for a fix to be developed, shipped and installed at most organisations that used the affected middleboxes. But today your Chrome (and presumably Firefox etc.) actually has anti-downgrade and any remaining middleboxes with that bug are just broken.

> Are the vendors going to give back your $1M? No they are not. Are they going to fix the appliance? Maybe.

This means that the organizations never learn plainly which of their purchases were jank and which were not, and do not improve their decision-making over time.

I think this disables/obscures an important market feedback mechanism. If a vendor sells you expensive crap with the implied promise that it is forward compatible/spec compliant, and the version field increments and, it turns out, your box isn't and breaks, it's likely to damage the reputation of the vendor (and perhaps the person within the organization that bought the janky device).

The game is an iterated one. Rewarding the not-diligent vendor/purchaser with diligent effort on the part of client software development teams does three things, none of which are desirable:

1. it shields shitty vendors from the market consequences of selling crap.

2. it shields shitty customers from the business consequences of buying crap.

3. it massively raises the barrier to entry for someone who wants to develop good and widely-accepted client software, keeping it in the realm of basically only a few large multinationals, none of whom really give a fuck about your privacy.

It's bad for everyone.

I think you've drunk too much Free Market kool aid and have begun to mistake a mere means for an end to itself. We're not trying to make the market set prices efficiently, we're trying to make the Network better, an efficient market is exciting for prophets of the dismal science, a Network that works better is good for all the actual people.

The network is better in the long run if people are more careful about which vendors they choose and vendors are disincentivized to ship broken shit.

Funnily enough, I actually suggested we adopt something like GREASE for OAuth parameters: https://mailarchive.ietf.org/arch/msg/oauth/Rqxs-QDqqdQkt36S...

There wasn’t a lot of support.

That list archive suggests that although there wasn't a clamour of applause for the idea at least some people were receptive. However it also suggests that there were plenty of more subtle incompatibilities which GREASing just this one place wouldn't solve.

It also seems like you admit that in other places ForgeRock is also doing something that's unsound:

> (Basically there are quite a few clients that use JSON mapping tools with enum types - List<JWSAlgorithm>. I know there are parts of our own codebase where we do this too).

Well like any large codebase there will be things that need improving. The handful of places where we have similar issues are relatively unlikely to cause a real problem in the near future.

Windows once used a similar fuzzing technique[1] to mitigate a video driver bug.

[1] https://devblogs.microsoft.com/oldnewthing/20040211-00/?p=40...

If you're designing things where you expect this to be an issue you can do a few things to ease your path:

Firstly you want to have a clear distinction between "Things that it's fine to just ignore because you don't understand them" and "Things that if you don't understand them that means you're screwed, abort". For example in PNG some of the bitflags cunningly concealed in the capitalisation of the extension chunk names indicate whether it's OK to just ignore the chunk if you don't understand it, and whether it's OK to just copy this chunk across into a modified image without understanding it. In X.509 there's a bitflag that means "If you don't understand this part of the certificate then you mustn't rely on the certificate at all".

Then you need a way for people to agree on how to allocate the extensions. The IETF has an impressive array of different mechanisms available, from "You need to actually write and publish a Standards Track document" to "First Come First Served" which suit different purposes. Consider just copying some or all of them. https://tools.ietf.org/html/rfc8126

TLS had big problems with (non-)interoperability and as a security feature couldn't just shrug and put up with never moving forward so it had to endure painful upgrades each time. This time they got GREASE (RFC 8701) to try to stop this ever happening again. Basically they agree how to extend things, then, reserve some of the extension namespace and routinely shove crap into it on purpose. Things that properly understand how extensions work will go "Eh, I don't understand this" and ignore the crap. Everything else explodes, and you catch the explosions early because they happen everywhere all the time with all the GREASE you're using.

GREASing new features has become a go-to strategy and you should consider this too. For example Encrypted Client Hello (the current iteration of ESNI) is intended to be used for all connections. Basically every server you talk to, you always say "Also here's an encrypted hello". Maybe the server can read it and the plaintext hello was just a dummy - maybe it just reads the plaintext hello, an eavesdropper can't tell. So if they want to block encrypted hello, they have to block everything, even though probably very few servers will offer this feature at first.

While I respect sticking to your guns, I don't think we're putting that genie back in its bottle. Maybe a reasonable middle ground, like a less-fraught version of the HTTPS redirect, would be to have any requests containing unknown query parameters redirect to the URLs with those parameters stripped. It does nothing to deter the people slapping them on there, but at least your visitors don't spread that plague by sharing your URL.

Of course, it all comes with the problem of needing to define up-front which query parameters your app accepts. Easy in a small app, wildly impractical in a large one with plenty of legacy code.

That's exactly what the 301 redirect is supposed to do. "Moved permanently."

it also breaks the ad and click tracking that those parameters are for in the first place. You're probably happy to do that, and that's just fine.

Better yet, send people to a trampoline page saying "You have followed a broken link. Did you mean to visit ...". Let people get the content, but rub their face in the fact that Facebook is breaking things.

What is that even supposed to accomplish other than signalling to the visitors that a) you dislike Facebook, b) you don't mind punishing others for what you perceive to be Facebook's wrongdoings? Make them abandon Facebook? Make them make Facebook to do things your way?

It probably wouldn't have any effect on Facebook. But if every link from Slack was hitting a page like this, Slack would fix it, because it makes them look stupid to their technical audience.

> If a HTTP request has unexpected and unsupported query parameters, such a GET request will normally fail. When I made this decision it seemed the cautious and conservative approach, but this caution has turned out to be a mistake on the modern web.

For a web page it might be a mistake. For an API it's often not a mistake, depending on the utility of the API. eg analytics reporting.

Git is another offender here. When they decided to switch to the smart protocol by default, the Git people decided that the way to probe for server support was to append a query parameter to the resource. Extra parameters are ignored by default on Apache (or something), and it worked on whatever the committer tested it on, so that became how Git worked. Whereas the parameter is part of the smart protocol, any server that supports it will give a response in line with what the smart protocol dictates, and the client will proceed with having recognized a server with smart protocol support. Dumb servers who ignore extra query parameters and respond with the kind of thing that dumb servers respond with will be treated as having no support for the smart protocol, and the Git client will allow the connection to fall back to the older protocol. Meanwhile, you can have a server that only supports the dumb protocol but doesn't ignore extra parameters, and so the switch in the client defaults just broke all transactions with those servers. The Git developers either didn't test it or didn't care. And something that makes the whole thing particularly silly/frustrating is that HTTP already has ways to do content negotiation which could have been used in lieu of the boneheaded scheme they came up with.

>Today's new query parameter is 's=NN', for various values of NN like '04' and '09'. I'm not sure what's generating these URLs, but it may be Slack

For a vague parameter like "s", my bet it's probably due to bad code rather than intentionally added. eg. someone constructs a URL but forgets to escape the components, and later (or downstream) some other programmer decides to "fix" it by using encodeuri (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...), which "works", but mangles the url

Actually, I think Twitter adds "s=NN" to their URLs when you share them depending on your device. It's something like NN=19 for Android, 20 for Windows, and 21 for iOS.

I REALLY hate that sharing URL. Makes me have to do another click to see the actual thread with replies directly. A subtle, but, to me, important difference.

"s" is a really dangerous parameter to use, though. 100% guaranteed that lots of sites already use it. I mean, fbclid isn't fun, but at least it's fairly unique.

IMO, query (search) parameters are originally designed for remote database search and forms. search criteria and form fields are often contain optional and conditional fields (conditional ie. only meaningful if some other field checked or having some specific value). but clients send them anyway because the logic (what is optional and which field depends on which) is known by the server. that's why client side programs consider server-side programs being lax accepting any parameter. however appending various cryptic 1-letter parameters without knowing the target http resource's semantics is unclever, since get parameters are part of the url, so changing them changes the url itself, so the new url does not necessarily points to the same resource. IMO, it's the web's originally visioned concept of identifier–resource-relation which is not adhered in today's websites. and this leads to dirty workarounds, unnecessary complexity and security holes.

The enpoint is possibly not the only adressee of the parameter. They may go to some middleware component. Or the JavaScript in the browser. So it makes sense to take parameters as messages that you ignore when unknown.

but then the middleware or js could just strip out it's extra params. Though that invites it's own problems - e.g. if the application actually used the `s` param, and something came along and started using it too... but this case is in general a world of hurt.

I'm not really sure why we'd want a world where the server doesn't know about all query args, but seems like we're already stuck with this.

Perhaps a general compromise is to generate a rel=canonical link [1] that only includes directly supported query parameters.

[1] https://en.wikipedia.org/wiki/Canonical_link_element

Well, I believe in Postel’s Law. Any opinions on the utility or danger of redirecting to the canonical URL instead?

E.g. I don’t want anyone’s campaign tracking tokens in the referer of our outbound links, or in URLs being shared around by copy-paste from the address bar.

However I also don’t want to inadvertently create a mechanism for someone to mechanically enumerate our routes (which we already see routinely attempted via return-path parameters), or walk into any similar trap besides.

I think this might be a good use case for history.replaceState - it'll change the URL in the address bar without actually redirecting.

Referenced (only partially quoted, and wholly unheeded) in the article, Postel's Law, also known as the Robustness Principle, says "be liberal in what you accept, and conservative in what you send."

https://en.m.wikipedia.org/wiki/Robustness_principle

Bypass the (proxy) cache: add a query parameter with a random value in it. Forgot, why I needed it but back in the day it was a big deal.

the old jquery would automatically add this parameter to your get requests to bust the cache.

I worked at a few marketing shops a while ago where we would manually increment a cache-busting query parameter any time we changed static assets (e.g., `/css/style.css?v=20100908`). We configured Apache to tell the client to cache these resources for a year, and this was a fairly common way of getting around that.

Forgetting to update this ended up be a common source of bugs.

Do those extra parameters break caches? The browser's cache doesn't know it's the same URL. Does Cloudflare? Akamai?

Yes and no. I believe all CDN have config available to put no query params, all query params or specific query params into cache keys. Whether people us that options is a different question.

Reading this article I start to wonder about how common parameter collision is in the wild.

Probably, your service doesn’t have any ‘utm_*’-parameters not intended but I could see ‘fbcid’ being used for other ids than just Facebook clients and ‘s’ is so generic that it must clash with thousands of services!

It's worth mentioning Postel's Law:

https://en.wikipedia.org/wiki/Robustness_principle

That said, query parameters on a GET are one thing, but unexpected arguments on CLIs are another, especially when the command can change/delete files, so I can see both sides of this.

I mean, that's almost the same debate as XHTML vs HTML5 of yesteryear...

Some people want strict requirements, but "whatever works for the end-user" usually wins.

...and as a result, we got monstrosities like user agent strings. But that's something we need to deal with I guess

I have a hard time understanding why one would care about such things. Just ignore additional query parameters you don't need and be done with it forever, what's the problem with that? Is it really worth writing a rant?

The reason is stated towards the end:

> In general, any laxness in actual implementations of a system can create a similar spiral of de facto requirements.

Those "de facto requirements" mean that writing something simple becomes inordinately complex because noone fixes broken clients. And you can't test changes because you don't have access to these broken clients.

It also leads to nasty bugs. For example, if you ignore extra parameters and provide defaults for others, now a misspelled parameter will silently fail.

The best example is how hard it is to write a browser. The standards are very complex, but it's magnified by the complexity of supporting "real," namely broken, HTML and CSS. That's contributing to the browser mono-culture.

Well, the alternative is that somehow you end with technology that no one bothers to use. Laxness allows to have multiple early, lazy implementations interoperate more or less correctly. Yes, in the later stages you end up with loads of technical depth and oligo- or mono-culture.

The logic is incoherent. Strictness creates requirements too.

The article isn’t arguing against requirements existing, it’s arguing against being bound to de facto requirements because you allowed a client you don’t own to depend on behavior you didn’t specify. (See Hyrum’s Law: https://github.com/dwmkerr/hacker-laws#hyrums-law-the-law-of...). By limiting the scope of unspecified behaviors, you also limit the scope of what makes for a de facto breaking change.

Something similar to URL query parameters is cookies. Most servers will ignore unknown cookies, but not always. The Great Suspender Chrome extension inserted tons and tons of cookies into various domains for some users, and there were so many cookies that this broke Google sites for some users.

https://github.com/greatsuspender/thegreatsuspender/issues/5...

Reply on news.ycombinator.com