Hacker News Reader

Comment by charcircuit | Hacker News Reader

Since domains are UTF-8 doesn't the assumption of ASCII breakdown?

Domain names are limited to a small subset of ASCII; the DNS protocol doesn’t support UTF-8.

To work around this, the international domain name standard defines an encoding called Punycode which maps Unicode to the limited character set DNS supports. The server is unaware of this, and so this optimised tolower() implementation works without any Unicode considerations.

Lower case is locale-sensitive, however. For example, tolower('I') in Turkish should be 'ı'.

And within unicode, doing it in a "dumb ascii" way probably needs some normalization of diacritics. Eg. 'é' should be U+0065 U+0301 ("e\xcc\x81"), not U+00E9 ("\xc3\x89").

Not sure how punycode handles this, I did once look deeply into it but that was years ago.

Punycode normalizes the case and other Unicode stuffs with the PRECIS Framework [1] before the label ever gets encoded to an ASCII-Compatible Encoding---those get prefixed with `xn--`. The root zone and top-level domains have an additional set of script-specific restrictions as well (RZ-LGR [2] in particular).

[1] https://datatracker.ietf.org/doc/html/rfc7564

[2] https://www.icann.org/resources/pages/root-zone-lgr-2015-06-...

Can you link me to where thin is specified. From what I can tell there is no requirement to use ASCII in domain names.

Section 2.2 and 2.3 of RFC 5890 [1] has a good overview of the pre-IDN state of things, with reference to where these were originally specified. In particular, the definition of 'LDH label' describes a single component of a traditional DNS name:

   This is the classical label form used, albeit with some additional
   restrictions, in hostnames [RFC0952].  Its syntax is identical to
   that described as the "preferred name syntax" in Section 3.5 of RFC
   1034 [RFC1034] as modified by RFC 1123 [RFC1123].  Briefly, it is a
   string consisting of ASCII letters, digits, and the hyphen with the
   further restriction that the hyphen cannot appear at the beginning or
   end of the string.  Like all DNS labels, its total length must not
   exceed 63 octets.

I misspoke slightly in my earlier comment: it turns out the DNS protocol does allow octets of any value in a label, but the Internet domain name system -- including all the existing clients, servers, registrars, etc -- as a whole does not. Which is part of why we need a fairly complicated set of specifications to make non-ASCII domain names work.

[1] https://datatracker.ietf.org/doc/html/rfc5890#section-2.2

DNS labels are limited to some ASCII characters. UTF support in DNS is available via punycode which is an encoding from UTF onto that restricted ASCII set acceptable for DNS. Libraries as the ones discussed here typically perform UTF to punycode conversions before doing any label comparisons to ensure accuracy. In particular this tolower implementation would likely be used against the punycode encoded version of the UTF domain.

https://en.wikipedia.org/wiki/Punycode

Absolutely. This article is reminiscent of the type of brief tutorials from the mid/late 90s in QBASIC newsletters introducing how to use ASM to get speed gains, with all the associated assumptions. The concept of "lowercase" breaks down terribly once you go beyond simple A-Z. You can apply similar rules to certain limited code sets, but a universal tolower() won't work so well with this method.

"Universal tolower()" doesn't even make sense. What would you expect from tolower("Σ"), "σ" or "ς"? If you assume tolower(toupper(mychar)) == tolower(mychar) then your code is going to break.

yes that is my point

Reply on news.ycombinator.com