Hacker News Reader

You can only type ~27% of my name with just ASCII (and even then one letter will not be exactly)... and i agree with you. If anything i'd go a bit further and say that, sure, use Unicode in places where you can find arbitrary text like documents, messages, etc but anything that has to do with the 'guts' of the computer should stay away from Unicode (or at least treat it as data, like how filenames are treated on Linux).

I disagree with the getting everyone on the Unicode bandwagon though, IMO Unicode has introduced a ton of problems exactly because it tries to be a ton of stuff at the same time. I don't know how exactly a better solution would be but i have a very hard time accepting that such a convoluted and error prone system is the best solution. IMO if decades later there are still issues with getting it right then there is something fundamentally wrong with the system itself and not with the applications and developers trying to work with it.

An existing working solution, even if not perfect, patched, with lot of baggage and technical debt is infinitely better than a non-yet invented ideal, perfect solution.

And even if the perfect solution existed right now, in a few decades it will be as filed with baggage as the current one.

Sometime one has to realize that hard problems are hard.

> Sometime one has to realize that hard problems are hard.

And trying to have a single thing solve everything doesn't help at all. When a problem is hard you try to break it into subproblems.

Unicode simply tries to do a lot of things at once.

Adding a variable decorator/annotation like @Unicode(german,french) would be a good stop-gap. You could only use ASCII characters unless you specified the script that you want to use. One could even set a max limit on how many scripts per variable. Because while I have used German characters in variables before (only if I'm referring to some law or spec), I never had a use case for more than 2 scripts within one variable.

The multiple scripts per variable thing is implemented in Rust via a lint. For the explicit enabling of single scripts, I have suggested that for Rust, but sadly people preferred allowing all identifiers (while giving an option to only have ascii but I'd argue this is unfair for anyone who only wants to use a specific non-ascii language, why do they have to suddenly allow all languages in their code base?). There are also practical concerns, like who says what a language is, which characters it contains, how that language is called, etc? Someone has to maintain all these lists.

> who says what a language is, which characters it contains, how that language is called, etc?

The Unicode Consortium already maintains all of that data in the CLDR (Common Locale Data Registry).

I think this is a good idea because once in a while you need to write non-ascii characters in names.

This mostly comes up when implementing tax rules or government administrative divisions as some countries have names/concepts which have no good translation into English, so you are left with using the non-English name, which often contains non-ASCII characters.

For your information the relevant Unicode specification is the Script_Extensions property [1]. (You can't easily filter by languages, so you should filter by scripts.)

[1] https://www.unicode.org/reports/tr24/tr24-32.html#Script_Ext...

The issue with this is less that this is possible and more that a lot of javascript ends up in production without ever getting compiled, linted, type-checked, etc. Stuff like this is designed to bypass what little human oversight there is to prevent bad things from happening. What is actually visible also depends on what fonts you have installed on your system. So, it's less clear cut than you think.

The problem is not so much that humans can't see this but that they are not looking very hard to begin with (otherwise, they'd be using the appropriate tools) and that we should rely less on them actively looking. Blind trust that things will be fine is the root problem here.

> The problem is not so much that humans can't see this but that they are not looking very hard to begin with (otherwise, they'd be using the appropriate tools) and that we should rely less on them actively looking.

And simply not allowing non-ASCII identifiers in the first place would be a move in that direction. Now you have one thing less to look for.

> But can't we all just sit down and admit that allowing non-ASCII characters in programming-language identifiers was a bad idea?

It's a bad idea only if all members in your team can easily produce and comprehend an ASCII-only code.

> Having invisible characters represent executable code is not just a dumb a idea, it's so hazardous that you might call it borderline malicious.

Not if those invisible characters do affect the rendering. Invisible formatting characters like ZWJ and ZWNJ are allowed because they are used in some scripts. The relevant Unicode specification [1] even provides a guideline to limit ZWJ and ZWNJ strictly to the context where they do affect the rendering.

That said, the Hangul filler and half-width Hangul filler were mistakes. They are purely legacy characters and never have been used in practice, so I encourage new languages to exclude them from the default (X)ID_Start/Continue set (Unicode can't do that because of the compatibility, maybe they can introduce another pair of properties without those characters).

[1] https://unicode.org/reports/tr31/#Layout_and_Format_Control_...

> The relevant Unicode specification [1] even provides a guideline to limit ZWJ and ZWNJ strictly to the context where they do affect the rendering.

Which is exactly what I am suggesting by saying non-ASCII characters should be banned from being used as identifiers, not from being present in the code-file all together or in the form of strings, etc.

If the formatting of your output in your applications (as seen by the user) depends on the names you've declared your variables with, then you are doing something horribly wrong.

You seem to think those formatting characters as something that should be in the higher-level protocol like HTML. They are not. They are used when two consecutive abstract characters can be combined in two or more different ways. And those different renderings frequently have different meanings. That's why they can't be simply removed when normalized; doing so will destroy the text.

We seem to be talking past one another. What Id like to see banned is non-ascii in identifiers, variables-names and nothing else.

While you respond as if I want to banish anything non-ASCII from all parts of all code-files except from HTML-templates. That’s certainly not what I’m advocating.

The following is IMO perfectly fine:

    var greeting = “Hello  (cowboy emoji)”;

The following is IMO not:

    var (emoji) = “Let’s party!”; // note identifier contains non-ascii

Do you still disagree? If so, can you outline why?

Okay, I think I see where you got confused. There are multiple levels of Unicode identifier support and you are probably not aware of all possible levels. Those levels are:

1. Identifiers can contain any octet with the highest bit set. Different octet sequences denote different names.

2. Identifiers can contain any Unicode code point (or scalar value, the fine distinction is not required here) above U+007F. Different (but possibly same-looking) code point sequences denote different names.

3. Identifiers can contain any Unicode code point in a predefined set, or two if the first character and subsequent characters are distinguished. Different code point sequences denote different names.

4. Same to 3, but these predefined sets derive from the Unicode Identifier and Pattern Syntax specification [1]---namely (X)ID_Start/Continue.

5. Same to 4, but now identifiers are normalized according to one of the Unicode normalization algorithms. So some different code point sequences now map to the same name, but only if they are semantically same according to Unicode.

6. Same to 5, but also has a rule to reduce unwanted identifiers. This may include confusable characters, virtually indistinguishable names and names with multiple unrelated scripts. Unicode itself provides many guidelines in the Unicode Security Mechanisms standard [2].

Levels 3, 4 and 5 are most common choices in programming languages. In particular emojis are not allowed for 4, so your example wouldn't work in such languages. For example JavaScript is one of them so `eval('var \u{1f600} = 42')` doesn't work (where U+1F600 is a smiling face). Both Python and Rust are at the level 5. Possibly unexpectedly, both C and C++ are at the level 3. Levels 1 and 2 are rare especially in modern languages; PHP is a famous example of the level 1.

Level 6 is a complex topic and there are varying degrees of implementations (for example Rust partially supports the level 6 via lints), but there is a notable example outside of programming languages: the Internationalized Domain Names. They have very strong constraints because any pair of confusable labels is a security problem. It seems that they have been successful in keeping the security of non-ASCII domains on par with ASCII-only domains, that is, not fully satisfactory but reasonable enough. (If you don't see the security issues of ASCII-only domains, PaypaI and rnastercard are examples of problematic ASCII labels that were never forbidden.)

I argue that the level 3+ is necessary and the level 5+ is desirable for international audiences. The level 5 would for example mean that `var 안녕하세요 = "annyonghaseyo";` (Korean) is allowed but `var (emoji) = "oh no";` is forbidden. I have outlined why the former is required in the last paragraph of [3]. Does my clarified stance make sense to you?

[1] https://unicode.org/reports/tr31/

[2] https://unicode.org/reports/tr39/

[3] https://news.ycombinator.com/item?id=29170954

To be clear I’m completely oblivious to what Unicode identifiers are. As such I’m not talking about them, and they are out of scope wrt to my point.

What I am advocating is that identifiers used for symbols in the programming language (variables-names, function-names, class-names, etc), should be strictly ASCII-based.

That’s simple, understandable and should be a sane default anywhere.

My opinion is that since nobody without a doctorate in Unicode actually fully understands Unicode, having a rule-set for identifiers built on top of the already bewildering Unicode rule-set is a sure-fire way to engineer for unexpected consequences and/or security issues.

Sure. Allow it if you must. But you must opt in to use it. It should be a non-default feature everywhere where it’s available.

> That’s simple, understandable and should be a sane default anywhere.

This is an usual canned reason given to reject any internationalization efforts, and it is likely only "simple, understandable" and "a sane default anywhere" to people like you. As you didn't give why they are simple, understandable in general, I don't see how your arguments are universally applicable.

> My opinion is that since nobody without a doctorate in Unicode actually fully understands Unicode, having a rule-set for identifiers built on top of the already bewildering Unicode rule-set is a sure-fire way to engineer for unexpected consequences and/or security issues.

That can be said for about all security issues, not just Unicode. That doesn't make you to avoid writing anything, does it? For the record, it is a valid choice to not write anything, but we normally exclude that choice when we are talking about the technology. And the "bewildering Unicode rule-set" is an one-off thing, as it is not like that Unicode produces incompatible standards every year. (Python 3 adopted Unicode identifiers 14 years ago [1] and implementations never changed, only underlying databases have been updated.)

[1] https://www.python.org/dev/peps/pep-3131/

Rust has a lint against this kind of attack + you can explicitly disable non-ASCII identifiers if you really want to

Ideally that lint would be on by default though. Most code doesn't use non-ASCII identifiers. It's not happened though because of uhm. political reasons.

Most code made by English speakers contains English word and Latin characters, so other languages and alphabets must be abandoned, and their native speakers must imprisoned until they understand their mistakes.

Abugidas and logographies banished

OK, OK, we can start with a warning in the compiler that use of any language except English is unsafe.

The lint is on by default in the latest version of the compiler

Yeah, let's just switch to Cosmopolitan Identifiers: https://obua.com/publications/cosmo-id/3/ :-)

But yeah, it would break existing code, sorry.

a lot of software used in shipping/logistics predates unicode