Hacker News Reader

However, the JS gettters are all over the place...

Given this:

    a = document.createElement('a')
    a.href = "/ö/o?ö/o#ö/o"
    console.log(a.hash, a.pathname, a.search)

we get:

    Browser  .hash       .pathname   .search 
 
    Chrome   #ö/o        /%C3%B6/o   ?%C3%B6/o  
    Firefox  #%C3%B6/o   /%C3%B6/o   ?%C3%B6/o  
    Safari   #%C3%B6/o   /%C3%B6/o   ?%C3%B6/o  
    Edge     #ö/o        /ö/o        ?ö/o  
    IE11     #ö/o        ö/o (ouch)  ?ö/o

Likewise for domains, some UAs return them Punycoded (but don't provide a decoder), some in Unicode, and PhantomJS passes it percent encoded...

It's a big mess.

I discovered this while working on a SPA router that can handle routes in either `pathname`, `search` or `hash` mode... The routes are defined as the keys of a JS object, in Unicode in the source, and we have to juggle back and forth to match what `document.location.*` returns.

It looks like Developer tools also use these JS utilities, so you can't trust Network tab when checking what URL is sent to a remote server; to check what's being sent one have to either inspect network traffic or check it server side.

In theory, this is specified in section 3.1 of https://www.ietf.org/rfc/rfc3987.txt (including the "MAY" section beginning "Systems accepting IRIs MAY convert the ireg-name component of an IRI as follows"). That covers the distinction between the domain and the rest of the URL.

The use of the page encoding is part of the browser behavior, not any RFC. It has been retroactively specified in https://url.spec.whatwg.org/#url-parsing

Thanks for the references!

> For URLs entered to address bar it seems that UTF-8 is used, not page encoding.

Maybe I'm misunderstanding, but how could a browser use page-encoding before the page has been fetched?

It uses encoding of a page URL is extracted from, not of a page URL leads to.

That's how: https://html.spec.whatwg.org/multipage/syntax.html#determini...

And how would you deal with an URL to the HTML itself?

There are 3 ways to get an URL:

a) URL is entered to the address bar. Default encoding (it seems UTF-8) is used for query and fragment. b) User clicks on a link. Page URL is used; page encoding is already known. c) Redirect to a new URL happens. I can't say out of my head what happens in this case.

This is just a recipe for massive confusion though.

You're saying if I click on a link with an IRI in a non-UTF8 html page, it means one thing.

But if I copy and paste that link into a text file, and later paste it into a browser, it'll mean something else entirely? Or if I save it to a bookmark using some service?

Yeah, it is a recipe for confusion. Browsers use a workaround: when you copy an URL of a link (using "Copy Link Address" in Chrome or "Copy Link Location" in Firefox) it copies escaped URL, not original URL.

> Query string and fragment are encoded using page encoding and then percent-escaped. For URLs entered to address bar it seems that UTF-8 is used, not page encoding.

Well, for links that appear in a page to be defined using page encoding doesn't sound so shocking. That doesn't explain why the path would use UTF-8 regardless of page encoding, though.

Are you sure URLs directly entered in the address bar use UTF-8 for query strings and fragments rather than system encoding? If my OS is set to use GB2312, how do URL fragments that I type into the browser get encoded?

A good question; I'm not sure UTF-8 is always used because systems we tested were all using UTF-8 system encoding.

This question was not relevant for Scrapy because it makes sense to use UTF-8 as default regardless of what browsers do when page encoding is unknown; unlike encoding of URLs extracted from HTML this UTF-8 default doesn't lead to incompatible behavior.

The not making sense bit is a consequence of several waves of changes, some of them not really involving standards.

The first thing to be added here, I expect, was the handling of the query string. This was needed to allow non-ASCII things to be submitted via forms; the need for this arose before there was any support for non-ASCII domain names, and likely before there was any real need for non-ascii paths.

The issue with forms as they were initially created is that they would submit to some server and the server would then process the query string. In doing so it would do whatever it did for non-ASCII stuff; there was no standard. In practice, it would percent-unescape and then treat the bytes as being in whatever encoding the server developer happened to default to. Typically this was the encoding the web page was in as well. Yes, this is totally busted if your form has a name field and the name being input is not representable in ISO-8859-1 or whatever you authored your page in, but this was an incredibly common way to handle non-ASCII in the 90s. This all predates my involvement in browsers, so I'm not sure which is the chicken and which is the egg here, but the upshot was that browsers started sending the query string in the page encoding (with various hacks that were not interoperable for a while in cases when the text was not representable in that encoding) and servers started depending on this behavior. Then the accept-charset attribute got added to <form> to allow overriding the default behavior for people who wanted something else. The behavior of form submission via the query string in terms of encodings is specified at https://html.spec.whatwg.org/multipage/forms.html#picking-an... and https://url.spec.whatwg.org/#concept-urlencoded-serializer and https://html.spec.whatwg.org/multipage/forms.html#url-encode... which bridges them.

I think the next step was url paths; for those browsers were inconsistent for a bit about whether UTF-8 was used or whether the page encoding was used, but eventually all aligned on UTF-8, and this got standardized in the HTML spec as well, I'm fairly certain.

And for hostnames, punycode encoding was more or less a must for doing DNS resolution. And at that point that ended up getting sent on the wire as well. https://tools.ietf.org/html/rfc2616#section-14.23 defines the Host header, which is presumably what we're talking about here, since that's the only way the hostname is sent to the server (note that this is NOT part of the same byte string as the path and query). Following the breadcrumbs for that through https://tools.ietf.org/html/rfc2396 it all talks about this header sending the DNS name, which is why I assume browsers settled on sending punycode here. %-encoding in hostnames wasn't really a thing for a while, I think; e.g. Firefox didn't even support it until about 6 months ago.

nice, this explains it!

That's some pretty useful documentation there. I hope that it at least forms part of a submission to the WHATWG folks.