Hacker News Reader

Comment by lizmat | Hacker News Reader

> Python 3 and Perl 6/Raku are worse for shell-like tasks than their predecessors! Mainly because of startup time and the string abstaction

Startup time can always be better, agree. But full support for Unicode is needed in this day and age, and that brings overhead, whichever way you do that, and especially so if you want to do it 100% correctly. If you're still living in an ASCII world, then by all means, go for it.

Additionally, in this world of source version control, and virtually unlimited diskspace for source-code, I think using scripts, rather than one-liners, will help better in maintainability rather than using one-liners.

View on HN

"Full support" for Unicode is almost never what you want unless your program literally interacts with font rendering, and then it's not even close to enough. Most of the time the important requirement is that you don't mangle data as it passes through your program, which e.g. early py3 spectacularly failed and is still not better at than py2 in the very common case where your input is not actually necessarily well-formed Unicode, but just an unrestricted byte array that by convention usually contains utf-8.

Most of what people hope for Unicode support to achieve beyond that (case-folding, homoglyph elimination, &c.) is not only not achieved, but not even possible.

Oil has better Unicode support than Python 2 or 3 for shell-like tasks, because of the way file systems, libc, and the kernel work.

Explained here:

http://www.oilshell.org/blog/2018/03/04.html#faq

which links:

http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ (by Armin Ronacher)

The summary is that there are two kinds of Unicode support:

- UTF-8 based: Go, Rust, and Oil (and Perl 5 it seems, not sure about Raku)

- array-of-codepoint based: bash/zsh/libc, Python, Java, JavaScript, Windows (JavaScript notably requires surrogate pairs, Python used to have build time configuration, now has complex storage heuristics, etc.)

This isn't a theoretical problem -- the Unicode problems in the other comment thread I linked are real and show up in practice.

Unicode support of Raku is grapheme based (NFG or Normalization Form Grapheme). The unicode theory: https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... , about the implementation: https://6guts.wordpress.com/2015/04/12/this-week-unicode-nor...

Perl (5) is array-of-codepoint based, at the logical level. Those codepoints might be internally stored as their encoded-to-UTF-8-bytes, or they might not, but this does not affect the usage of the string.

Many don't really understand the string model (because many don't really understand character encoding) but it comes down to: all input and output is going to be bytes, which by default is stored as the codepoints sharing the ordinals of those bytes, and there are several mechanisms by which you can manually or automatically decode/encode those byte ordinals to the represented characters; for most text processing, you do this on STDIN and STDOUT.

Reply on news.ycombinator.com