Hacker News Reader

Comment by lorenzfx | Hacker News Reader

> But again, the population that will be affected is the 10% who deal with Unicode.

I doubt that only 10% of people are dealing with unicode. Rough estimate 10% of the world speak English as a first language and almost all other languages are written with (at least some) non-ascii characters.

In my experience (I'm from the 90% part of the world), people new to programming and python stumble really early into some (to them) hard to debug (UnicodeDecodeError: ... can't decode byte) errors in python 2. Then it's time to explain the whole strings are not bytes, unicode, encoding and so on thing. In python 3 that's also needed, perhaps even earlier, but bugs are much easier to fix and programs fail on the first run, not when you finally enter some non-ascii characters.

In my mind, python 3 clearly is superior to python 2, and would be even if nothing but the unicode support changed.

View on HN

Actually, I think the strictness of python3 with regards to bytes/strings is particularly useful for people who speak English as their first language. python3 will force them into writing software that can work with international strings whereas python2 code written by an English speaking person would in many cases just be broken for the international users.

And believe me, as an "international" person, this is profoundly annoying and has trained me in avoiding using the non-ASCII letters of my language which of course "castrates" the written language somewhat (i.e. When people use non-ASCII letters for folder names I think they are just asking for trouble - how normal is that?)

I love how upfront python3 is with the difference between raw bytes and strings. I work a lot with python and in particular I do a lot of work with serializing and deserializing stuff from/to binary blobs. Due to platform issues that I hope will change in the next year or so I can't use python3 currently but I keep all my code compatible with both and all tests should run with python2 and python3. Yes, I am firmly in the python3 camp :)

pycharm inspection definitely helps with keeping code compatible with 2 and 3.

> When people use non-ASCII letters for folder names I think they are just asking for trouble

Well they are

> how normal is that?

Not as normal as it ought to be.

Folder names (and file names) are identifiers. Identifiers should identify. If you allow unicode characters, then you can have two character sequences that look the same but are actually distinct. This is confusing at best and at worst (in URLs) could facilitate fraud.

> If you allow unicode characters, then you can have two character sequences that look the same but are actually distinct.

That would be a software bug. If you want to compare unicode strings, you need to normalize them first following the rules laid out in the standard. https://en.wikipedia.org/wiki/Unicode_equivalence

git for example fails that test. (try creating a repo with a file named 'ü' and check it out both on a mac and a linux system)

There are more issues about glyphs that are a distinct character but look the same in a given font, but what's your proposal? All people transliterate everything to ascii? Display punycode in URLs?

> Identifiers should identify

Except that, without Unicode, you now have to use a slighly restricted subset of many European languages and cannot use Cyrillic, Chinese etc at all. So people who are not operating in an English environment are limited in their ability to clearly identify things.

"then you can have two character sequences that look the same but are actually distinct."

If you accept that argument, you should also remove either 1 or l, O or 0, S or 5 etc.

> Rough estimate 10% of the world speak English as a first language

ASCII isn't even good enough for that 10% – British people need £, which isn't in ASCII – not to mention things like em dashes, en dashes, and curly quotation marks that word processors tend to add.

They're in latin-1 though, which is a byte-per-character encoding.

I don't think anyone who deals with character encodings should encourage latin-1 in 2016 instead of utf8. Fair enough, your old C code will work, but everyone else will hate you.

Except, of course, if you deal with Windows...

.. in which case it's Windows-1252 rather than latin-1. And maybe you have to deal with UCS-2, god help you.

Except that encoding depends on your regional settings, which in Baltics is Windows-1257 ;)

Kids these days also use emoji.

Also unicode.

http://unicode.org/emoji/charts/full-emoji-list.html

Emoji have been part of Unicode since 1993 (e.g. ️)

I loved python2, but now that I'm in a company where everything is python3 I probably save a few days a year of not mucking about with Unicode BS. It's wonderful. There may be other advantages but even if it were JUST "python2 without horrible Unicode idiosyncrasies" it would be worth it.

> I doubt that only 10% of people are dealing with unicode. Rough estimate 10% of the world speak English as a first language and almost all other languages are written with (at least some) non-ascii characters.

I believe he meant people that have to actually deal with complications of unicode/bytestring themselves.

For most uses, you just handle "strings", and don't really care if they're unicode or not. Only lower-level libraries/apps will ever care if these strings are unicode or not.

Yup, that's what I meant!

> I doubt that only 10% of people are dealing with unicode.

I will use a single character to express my absolute agreement:

Edit: I guess even Lisp developers have to consider Unicode.

True.

Even if your Lisp supports unicode, it does not make a web framework/application support unicode just so.

Disclaimer: I've added UTF-8 support to a Lisp-based web framework, on top of a Lisp implementation supporting Unicode.

> Edit: I guess even Lisp developers have to consider Unicode.

Why wouldn't they?

To explain/spoil the joke: HN does not support most of Unicode and is written in a Lisp variant.

Thanks, I clearly didn't get it :-)

Does anyone know what underlying differences there are in Python 3 unicode handling versus Go's rune system. I have found the latter far easier to work with when dealing with text characters in other languages.

Reply on news.ycombinator.com