Arc's Unicode support (by the news.yc patch writer)
collison.ie
collison.ie
use utf8;
Filehandles aren't utf8 by default, but can be put into utf8 mode with the 'binmode' function. (You can go further, and tag filehandles with pretty much any charset encoding and you'll get the Right Thing happening with reads and writes).
Other data sources (db handles etc) generally have some API to control this too.
As a convenience -CIO option to perl puts stdin and stdout into utf8.
Defaulting to off is just the price of back-compatability. Not sure what the plan is for perl6 in this regard.
$ cat > foo.py
# -*- coding: <utf-8> -*-
# even with the above character encoding
def ô():
return "ô"
$ python foo.py
File "foo.py", line 3
SyntaxError: Non-ASCII character 'xc3' in file foo.py on line 3, but no
encoding declared; see http://www.python.org/peps/pep-0263.html
Q Do people use Unicode identifiers in their source code?
[0] BDFL made this as a design choice to default to ascii.
powerbook.local 104> cat > foo.py
# -*- coding: <utf-8> -*-
def ô():
return "ô"
powerbook.local 105> python foo.py
File "foo.py", line 2
SyntaxError: Non-ASCII character '\xc3' in file foo.py on line 2, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
powerbook.local 106>
But even when the encoding is set properly python still complains:
powerbook.local 106> cat > foo.py
# -*- coding: utf-8 -*-
def ô():
return "ô"
powerbook.local 107> python foo.py
File "foo.py", line 2
def ô():
^
SyntaxError: invalid syntax
powerbook.local 108>cat > foo.py copies standard input into the file foo.py. The rest of the characters you're seeing in the message are what I typed in. The only character you don't see is the Ctrl-D I type to indicate EOF. This is a quick and easy way of creating foo.py without using an editor.
True this syntax erases the file. But all I did was cut+paste the example from the original source & added the source code encoding on posting. I knew the example code missed the encoding and I checked the example text for extra unicode characters (another possible source of errors) as I saved to the file & checked with cat. So I tried the example on the CLI before I posted and overlooked the redirection. It's not worth lying, way too many sharp people to pick you up. Just a simple display mistake.
Btw, the inference I'm lying. Does that also apply to the original author as well?
"... Some time later ..."
Doing a bit of digging. It appears the 'cat > foo.py' works like this ...
"... Redirection of output causes the file whose name results from the expansion of word to be opened for writing on file descriptor n , or the standard output (file descriptor 1) if n is not specified. If the file does not exist it is created; if it does exist it is truncated to zero size. ..." [0]
So my theory is you have tried this with a full file & it has erased itself. Thanks for the pick-up though. It is a fine distinction and something I've not come across before. We are both right but at different times.
Instead, he said: "I realize that supporting only Ascii is uninternational to a point that's almost offensive, like calling Beijing Peking, or Roma Rome (hmm, wait a minute). But the kind of people who would be offended by that wouldn't like Arc anyway."
There was no reason to take a swipe at the "kind of people" who care about internationalization, which is to say:
* people who are not mono-lingual anglophones
* people who want to build real-world applications
Why shouldn't people in those categories be interested in Arc? Why should they be excluded? And why treat it as a matter of political correctness rather than just a technological decision?
Paul threw the first punch and the blogosphere punched back.
Furthermore, his general tendency to divide and conquer the programing world in that way is why there was already a huge pool of haters ready to pounce on him. Every essay of his implies that there are people who get it and people who don't and one can distinguish between them by seeing which people agree with him and which do not.
He only needed to say: "Arc does not yet support Unicode" and the whole thing would have been avoided.
Why should PG say "unicode support will be easy and that he will accept patches"? Wouldn't making promises be contrary to the entire disclaimer he wrote? (Have you read disclaimers before? Paul's addresses at least as many facets as those written by lawyers.)
If bloggers want to influence PG into feeling guilty in order to get him to spend more time on arc regardless of his disclaimer of "when it's done, it's done," they can do it by phrasing things like adults. Or, writing their own languages.
(Great post. Maybe PG wouldn't have realized this perspective had you not posted it for users to upvote. I know I didn't notice your perspective. On the other hand, I wouldn't change a thing. You're expecting PG to plan for all potential hurt feelings, be it Mac users, Linux users, corporations, people in remote locations without access to the Internet because this means they can't download arc, and more. This is impossible to do, and leads to nothing being released at all.)
Simply take this paragraph:
"Which is why, incidentally, Arc only supports Ascii. MzScheme, which the current version of Arc compiles to, has some more advanced plan for dealing with characters. But it would probably have taken me a couple days to figure out how to interact with it, and I don't want to spend even one day dealing with character sets. Character sets are a black hole. I realize that supporting only Ascii is uninternational to a point that's almost offensive, like calling Beijing Peking, or Roma Rome (hmm, wait a minute). But the kind of people who would be offended by that wouldn't like Arc anyway."
And change it to:
"Currently, Arc only supports Ascii. MzScheme, which the current version of Arc compiles to, has some more advanced plan for dealing with characters. At some point (I don't know when) I or someone else on the Arc team will probably figure out how to take advantage of it."
That's all. Say less. Stick to the technology. Avoid politics. Controversy avoided.
(Btw. same thing with the controversy over presentational markup vs CSS. AFAICT the Arc framework don't do any layout at all out of the box, including any presentational markup.)
Now PG seem to believe that controversy is inevitable when creating a new language so you might as well throw a few random punches. However if you want to build a vibrant community around a language I think it is very important to choose you battles carefully and your swipes wisely.