Perl Unicode Cookbook | Hacker News Reader

Perl Unicode Cookbook

training.perl.com

58 pointsdraegtun22 comments

I love to develop with perl, but unicode-related stuff was always frustrating for me. Probably because I deal with it so rarely and everytime I have to look documentation to find out how to read unicode file, run regexps on unicode strings, etc.

I am wondering, how other major languages deal with unicode? Is there language which work with unicode files/text transparently without /me involving into setting something differently from defaults?

Perl comes out best according to Tom Christiansen (author of this Perl Unicode Cookook post).

See his Unicode Support Shootout - The Good, the Bad, & the (mostly) Ugly presentation at OSCON 2011 where he gave a comparison of Unicode handling in Javascript, PHP, Go, Python, Ruby, Java, and Perl.

ref: http://training.perl.com/OSCON2011/index.html

Perl is in the lower tier IMHO. Below Python and Haskell. Because while it has a good core unicode support, every single library makes different choices, so your code constantly needs to encode/decode. Javascript seems to avoid encoding issues entirely, at least in the browser, though Tom has some nice points about it not full supporting unicode. <http://kitenet.net/~joey/blog/entry/unicode_ate_my_homework/...;

Can you be more specific? Perl's utf8 support is IMHO the cleanest and most transparent. Generally libraries don't have to care at all about the encoding of the stuff they deal with. I'd be curious what trouble you've had.

(It's true that early versions of "use utf8" were horribly broken, munging strings silently at I/O time, but that was 10 years ago or more)

Nearly every XS library takes input with utf-8 encoded to bytes, so you have to manually juggle the encoding when using it. There is no documentation stating what encoding any library expects. I just checked a source tree and found 41 calls to encode_utf8 and 42 calls to decode_utf8 to deal with this across dozens of libraries.

Um... of course they do. XS is a translation library and the domain you're translating to (C) has no (well, no sane) native encoding handling. So you pass byte buffers. That's not a ding on perl though, it's an inherent impedance you get when you cross interface domains. JNI libraries tend to have most of the same problems, either forcing the user to do the UTF16/UTF8 conversion or doing it in the library.

I guess I'm surprised that Python and Haskell don't have equivalent messes when dealing with native libraries. Are you sure they don't?

Perl XS libraries often provide a full object oriented wrapper around the native C calls. They always have, at a minimum, a thin perl wrapper function. There's plenty of scope for the encoding to be done automatically, but the current state is that it's not even done manually by the libraries.

Haskell's equivilant FFI interfaces do encoding conversions, so that if they return a String, it's a sequence of unicode code points, as the Haskell standard requires. Those that return raw data will use an appropriate ByteString type. Haskell didn't used to do encoding conversions for FilePaths in the FFI, but this has been fixed in version 7.4, and is even done in a way that doesn't require unix filenames be utf-8 encoded -- other encodings will pass through unchanged.

Encoding conversions at the FFI layer would be very nice. How do the Haskell interfaces know which encodings to expect?

"every time you call a library or do IO you have to manually deal with conversions.." (from the blog post)

This is certainly not true for libraries; Only done that to workaround faults in other "systems"[1].

For IO, you don't have to "manually deal with conversion". But you do have to, if you want to do it correctly, tell it what encodings the files you're going to read are. But you do this once for all IO.

There's a reason for this, and it has nothing to do with the language: You can't detect encodings. So, you either standardize on one encoding to rule them all or supply it when doing IO. To be honest, I seldom do it the "correct" way, and it works fine most of the time.

[1] E.g., the DB table had one encoding, but the data was actually stored using another one.

Edit: Just for the record, dealing with this can be a pain in Perl. Keeping things backwards-compatible means everything is not as streamlined as it could be. But at least it works (I'm looking at you PHP ;)

As someone who munges Unicode data on (a far too!) regular basis I have never found Perl wanting and believe it to be the best tool for the job.

>Perl comes out best according to Tom Christiansen

It would be hard to find a more bias source.

It would be exceedingly difficult to find someone more fastidious and willing to perform exhaustive research. If Tom has a bias, it's because he knows of that which he speaks.

Perl is definitely bipolar in this respect -- I've never had an easier time of dealing with Unicode issues than in Perl, but I've also never cursed Unicode so much while dealing with it in Perl.

Could you say in what other languages you dealt with Unicode?

Python has native support for unicode strings, not Unicode support in native string types. Sigh, this is where it gets into bs world, because you have to figure out how to convert back and forth appropriately. Just thought I'd share my recent frustration.

There's something wrong with the font size of code samples.

code snippets tends to be long. so they have font-size: 65% for <code>

I've found increasing font is required anyway, since my browser default font size a bit small for my screen resolution.

Increasing font size is quite easy in most browsers. Hold Ctrl key and use mouse wheel to adjust size. This is per-site setting in most browsers I believe.

I've reenabled Firefox's minimum font size; it's a bit buried in the preferences (fonts->advanced) but it's more effective to fix sites where the sizes lack consistency. For most sites NoSquint works well.

this particular site uses browsers defaults. calling this lack of consistency is wrong :)

I'd love to have the POD of this locally, but can't seem to find it anywhere. Anyone know where it lives?

Isn't that just a little ironic that the Unicode bullets have been converted to the prescription symbol?

I'm pretty sure that's on purpose (Unicode prescription 1, prescription 2, etc.). Or did my <sarcasm> tag detector mess up again?

Reply on news.ycombinator.com