Fun with Unicode in Swift
tworingsoft.com
tworingsoft.com
1) As the article demonstrates, this allows obfuscation contests in easy mode. Strangely, the author claims:
> Nor do I think most of these tricks would get by a code review or go unnoticed using something as simple as syntax coloring.
How would syntax highlighting catch that? Those are valid identifiers! And yes, these "tricks" would almost certainly go unnoticed in a code review, at least when you don't assume that your coworker is maliciously inserting bugs.
2) Even in the absence of malicious intent, Unicode identifiers open the door to bugs and head-scratchers: different characters looking identical, or the same character encoded in different ways.
3) ASCII letters (a-z, A-Z), digits (0-9), and underscore are kind of the lowest common demoninator. When you allow Unicode identifiers, you don't "internationalize" your code, you're doing the exact opposite! If you're using ä, ö, ü, ß in your API, how is a Canadian supposed to easily use it?
When I was in france, all the software were written using french identifiers. It was much better this way since none of the code was meant to be shared outside of france and it made the code much more readable for people who were supposed to work on it. Why should it be possible only for latin based languages?
> it made the code much more readable
"Much more readable"? It made the code base inconsistent with the rest of the language. Are you saying that they were lacking even basic English language skills?
I don't see any problem with source code that says something like 'auto maßtab = waage.m_maßtab();' 'auto' and '=' are simply language tokens.
There's no potential for confusion if you allow ASCII letters and 'ß'. Confusion can arise, however, when you allow all Unicode letters.
To me, using English keywords everywhere is a historical accident, not a design goal. In next-generation systems like Subtext [1], it looks entirely possible to add localization to source code, too.
Once you throw out the Lisp way of having one namespace for all symbols (or, OK, like 7 in CL), to be used as functions or values, and go to a system where you want to be able to statically type-check everything, why not fully commit to it and let users give local names to identifiers? Then we won't be limited to legacy C-style naming rules, or have to argue about how many space characters to add to make all their names line up on a high-resolution VT100 emulator, either.
[1]: https://en.wikipedia.org/wiki/Subtext_(programming_language)
Yes i'm saying that a lot of colleagues were lacking basic english skills, and i don't see the issue with that. English is a nice to have to communicate with people abroad, but absolutely not necessary to live in France.
Oddly (to me, anyway), the software house that made the program was located in Africa.
Sometimes I'm tempted to use Russian identifiers for application domain terminology, because very often there's no adequate translation or I just don't know English enough to translate some terms. I like the following example: every man here has name combined from last name (Фамилия) which is used for all family members, first name (Имя), that's pretty standard. But there's also third name: Отчество, which is basically father's name with some suffix, e.g. father Иван will have child with third name Иванович. Now I've seen all kinds of translations for those terms, including plain wrong. Last name, family name for Фамилия, first name, name for имя, father name, second name, patronymic, patronymic name for отчество. I've seen bugs because one developer interpreted those terms differently from another.
And that's just for name. There are plenty of other terms. Add to that that many developers are bad at English and at best will use first result from Google Translate. So may be national identifiers have its place.
Sounds like you get to skirt a few of the problems outlines in Falsehoods Programmers Believe About Names[1]. That's nice for programmers in Russia that deal with Russia only, if true. :)
1: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
Of course the whole point of the article you mentioned is that making such assumptions leads to odd behavior. I (an American) lived in Russia for a while. One time I was having a couch delivered to my apartment and had to give the store my full name. I was delighted to see my middle name show up on the receipt in the отчество (patronymic) field with the Russian patronymic suffix ович attached. I still got the couch no problem, but I got a laugh out of their attempt to shoehorn my (clearly not Russian) names into their naming paradigm.
Unicode identifiers can be perfectly well-defined, and many languages have them. The general idea is an identifier can start with any character that has derived property XID_Start, and the remainder can be any characters that have derived property XID_Continue.
different characters looking identical, or the same character encoded in different ways
Follow the recommendation of UAX #31 and normalize identifiers to NFKC prior to comparing them. For example, the ligature variants in the article should not -- if recommendations are followed -- be different identifiers. Here's some Python (3):
>>> import unicodedata
>>> raw = ['vpnTrafficPort', 'vpnTrafficPort', 'vpnTrafficPort', 'vpnTrafficPort']
>>> normalized = set(unicodedata.normalize('NFKC', s) for s in raw)
>>> normalized
{'vpnTrafficPort'}
The real difficulty with Unicode identifiers is in places where you really can't avoid Unicode: user inputs. Those who don't read Unicode technical reports are doomed to suffer the moment they build, say, a user-account system that comes into contact with the real world.
When you allow Unicode identifiers, you don't "internationalize" your code
No, you let other people localize their code. We (the tech world) spent decades making everyone else learn English as a prerequisite to learning a programming language. Now we have the ability to lessen that burden and let the non-ASCII world (which does in fact include English!) spend less time learning English and more time writing code. We probably should do that, even if it seems icky to you.
Can you imagine what it would be like if every library had to be released in N different translations? Sounds like an exceptionally bad idea. A huge step back for our field.
But there are plenty of monolingual dev teams out there whose shared language isn't English and isn't written in ASCII. Why should they be forced to write code in English if nobody who works with it is an English speaker?
I already regularly see people post questions to Stack Overflow and mailing lists and IRC where names of classes and functions and variables are ASCII but not English (i.e., someone writes a class called "Utilisateur", not "User"). Why not let them just use their language fully?
English will continue to be the lingua franca of programming for the foreseeable future, and full professional competence will probably require being able to read APIs that are in English.
However, for the sake of learners everywhere, it is important that it be possible to program using non-ascii identifiers. Asking a 10 year old who doesn't speak English to learn what "while" means is quite different from requiring their variables to be in a language they don't understand.
char *speicher; // German for memory
(define f (keizoku) ...) ; Japanese for continuationYou're proposing that we replace the rules for identifier equality (comparing case-sensitive ASCII strings composed of A-Z, a-z, 0-9, _ is very easy to understand) with a complex system which hardly anyone can remember in its entirety, and even fewer people will want to learn? All so that according to you, some people need to learn a little less English? I sure hope that those normalization rules have been translated to lots of languages, otherwise those people would still need to learn English...
> Now we have the ability to lessen that burden and let the non-ASCII world (which does in fact include English!) spend less time learning English and more time writing code.
Do you seriously think that a programmer needs lower English skills just because they can use non-ASCII characters in their function names? That programming language's standard library is in English, its documentation is in English, external libraries are in English, their documentation is in English, StackOverflow is in English, but you need to "spend less time learning English" because you can name your variable "über_schwellwert" instead of "exceeds_threshold"?! Come on.
Of course you can keep using ASCII strings for your program but they should be able to name things however they like too.
But that's the point – you'll still need to learn just as much English when you have Unicode identifiers as when you only have ASCII identifiers.
Perl apparently handles this well: you have to declare that a file uses a particular script if you're using non-ASCII identifiers.
So, no calling functions in other files that use other scripts?
A reasonable restriction is to demand that all characters in any one identifier are from the same script, and that purely typographical devices such as ligatures are banned.
I actually am not quite clear on the details: one source made it sound like a limited set of scripts was enabled by default. Perhaps someone who knows Perl better can clarify.
use utf8 ("Greek", "Cyrillic");
That's insane. Not in my programming language (lawn, driveway, ..).
Other languages tend to solve this by stating that version X of the language uses identifiers having those properties in version Y of the Unicode database. This is a problem for C and C++ because of how infrequently those languages update their standards and how glacially slowly implementations adopt new versions of the standards. It's less of a problem for languages like Rust where the language spec and standard toolchain evolve on a much faster (at least one update per year) cadence.
The tweet you link complains about U+2800. That character is not in XID_Start or XID_Continue.
http://www.unicode.org/Public/11.0.0/ucd/DerivedCoreProperti...
So a language adopting XID_Start/XID_Continue would not allow U+2800 in an identifier. And in fact if I try it in Python I get "SyntaxError: invalid character in identifier".
I haven’t fully digested the links to the C working group, but as those Perl docs point out, you have to do some work even after restricting to XID_Start/Continue, even if you don’t have to forbid other characters.
The suggestion it cares most about from UAX#31 is to restrict to scripts that actually are in use (i.e., don't let people name variables using Linear B characters). If you want to layer that on top of a base pool of identifier characters taken from XID_Start/XID_Continue, you can, and that's not the same as "invent your own base pool of identifier characters".
The big thing you get from reading UTR#36 and UTS#39 is learning how to detect or prevent homoglyph attacks (like people registering "paypal.com" but with Cyrillic "a").
And UTS#39 takes the ideas from UAX#31 all the way and gives you an example of defining profiles on top of the base set of identifier characters to deal with specific issues. It's OK to do that!
Finally, the things that, in a programming language's allowed identifier syntax, would be prevented by going to a more restrictive profile on top of XID_Start/XID_Continue, are vanishingly rare and tend to be exploitable only if you're already completely owned. For example, if someone can slip a mixed-script confusable identifier into your source, they can already slip things into your source; you're owned in so many ways at that point that it starts seeming silly to focus with laser intensity on just this one issue. Which means that if you just want an easy-ish to implement baseline recommendation, XID_Start/XID_Continue isn't that bad.
Simply reject any piece of input, such as a user display name, which uses characters from two or more different scripts, or purely typographical devices such as ligatures.
Reject any mixed-script confusable (and for certain types of inputs, like email addresses, validate sub-components separately -- someone might get to choose their local-part but not their domain).
For strings which aren't mixed-script confusable, if you need to use them as identifiers and perform comparisons with them, normalize to NFKC and case fold. Doing so will eliminate any ligatures, stylistic variants, composed versus decomposed forms, or other things that people use to call Unicode "weird".
In Swift all identifiers are Unicode - basic letters are part of Unicode. The entire source file is Unicode all the way through.
> how is a Canadian supposed to easily use it?
Using their keyboard? Can't a Canadian type these letters in the exactly the same way as anyone else would?
> In Swift all identifiers are Unicode - basic letters are part of Unicode.
Yes, and it's an exceptionally bad idea to allow non-basic letters in identifiers, as the linked article demonstrates.
>> how is a Canadian supposed to easily use it?
> Using their keyboard? Can't a Canadian type these letters in the exactly the same way as anyone else would?
No, they can't. ä, ö, ü, ß are part of the German keyboard layout QWERTZ, but they're not on a QWERTY keyboard.
Anyone can type them using a modifier key. I have a UK keyboard and to get ß I hold option and press s. It's no different to using the shift key. If you've got no problem with capital letters you should have no problem with these letters.
Sometimes I boot into a machine where the keyboard layout is set to English. Finding the right keys on my non-English keyboard to even just type "ls -l /" or similar, is a major pain in the ass. It would be the same for a Canadian having to type 'ß' on an irregular basis.
Let people who have to interoperate with multilingual colleagues settle on conventions for doing so. Let people who don't use their own language already.
If you trust some third party enough to download code they wrote and run it, the added risk that you confuse a few identifiers and run code you didn’t intend to run isn’t that big.
If this happens in code you wrote, you fix it when you discover it or learn to live with it, just as when you find the same problems with ASCII identifiers. Those also can be confusing (Il0 vs I1O, color vs colour, misspellings, etc)
Also, inconsistent transliteration to ASCII can introduce problems. Using native strings can prevent that.
Author here, you're absolutely right. When I wrote this, I was thinking of some of the more obvious tricks like the alternate semicolon, parens or curlies. I'm gonna edit the article to qualify that, and add a note about maybe using a SwiftLint exclusion rule containing all these unicode characters.
Thanks for the discussion! :)
In dense scientific code there’s always been a place for special characters. (apl, anyone?) To the right kind of programmer, using the proper mathematical notation for operations and variables in a function can greatly improve readability and clarity.
Unicode for use in theorem proving languages also has a role — agda comes to mind here.
http://perl11.org/blog/unicode-identifiers.html
It's not fun, it's a risk.