Hacker News Reader

Comment by 21 | Hacker News Reader

I've seen this story happen twice before.

20 years ago there was this great MP3 player, WinAmp 2. And then they released WinAmp 3, which broke compatibility with skins and plugins and which was slow. People didn't upgrade. Finally, they came to their senses and they released WinAmp 5, marketed as 2+3, which was faster and brought back compatibility with older stuff.

In the retail trading Forex world, there is this great trading app called MetaTrader 4. Many years ago they released MetaTrader 5, which broke backward compatibility and removed some popular features. People didn't upgrade. Today, they are finally bringing back the removed features and making MetaTrader 5 able to run MetaTrader 4 code.

Me, I wait for Python 5 (=2+3) which will be able to import Python 2 modules, so that you can gradually convert your code to the newer version.

I have tens of thousands of lines of Python 2 code in a big system. I can't just take 2 months off to move all of it to Python 3. Moving it in pieces is also not really possible, since there are many inter-dependencies.

Uglyfying my code with the "six" module it's also not a solution, since when I'll move, I won't care about Python 2 anymore.

So basically I'm just waiting until a consensus emerges.

It should be noted that the quantity of Python 2 code keeps on growing, I wrote most of my code while Python 3 existed. If Python 3 allowed an easy path forward, we wouldn't be in this situation.

View on HN

All of the reasonable "2+3" features in Python have already landed, like u'' strings in Python 3.3. The really big issues, like the difference between strings and bytes objects, can't really be emulated away. This isn't like WinAmp or MetaTrader where you can just reimplement the old interface in the new application. Code that mixes strings and bytestrings freely is simply not going to interface well with Python 3 code.

There are few paths for upgrading. You haven't mentioned 2to3, but I will. I've migraded projects of the same size you mention (10s of kloc) with 2to3 and the process took days, not months. About three days with one developer, actually.

There are a number of projects which also have code bases which are compatible with both Python 2 and 3. You say that once you move, you won't care about Python 2 any more, but I that's not a good reason to reject six outright. In my experience, the code isn't really "uglified" but there are just a few minor cases here and there you need to think about.

Anyway. Small-ish projects like the one you are talking about are usually not as hard to migrate as you might think. There are some exceptions, of course.

I agree that the bytes/string problem is a tough one. But the dev team created it and dropped it into the laps of it's users.

The Windows API had the same problem, it was ASCII and then they added Unicode. But instead of just forcing your app to exclusively use a single one of them (sort of like use Py2 OR Py3), they allowed you to mix and match your code, and call either the old or the new version of the API. And gradually, people stopped using the older version and started using the Unicode one. Today, new Windows APIs are strictly Unicode.

Of course, the solution was ugly from their side, having 2 duplicate APIs for the same thing, but it didn't brought pain to their users and allowed them to proceed at leisure.

So, it's like Python. You can still mix and match; you just have to explicitly convert between them at the appropriate points.

The main breaking change is that you can't read a stream of bytes from a byte-oriented resource (say, a network connection that by definition can only send a stream of octets) and magically treat it like text elsewhere. While that's a tempting thing to want to do, it's been an unending fountain of bugs over the years. For example, in Python 2.7:

  >>> a = 'this is a test ಠ_ಠ'
  >>> b = bytes(a)
  >>> b
  'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0'
  >>> type(b)
  <type 'str'>

In Python 3.5:

  >>> a = 'this is a test ಠ_ಠ'
  >>> print(a)
  this is a test ಠ_ಠ
  >>> b = bytes(a)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: string argument without an encoding
  >>> b = bytes(a.encode('utf-8'))
  >>> b
  b'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0'
  >>> type(b)
  <class 'bytes'>

In 2.7, you encode a series of Unicode values back to... a string? This is the source of all kinds of fun as functions expecting a string as input don't have a way to tell whether it's already been encoded or if they need to do so. In 3.5, there's no ambiguity. Functions that write to files or network sockets can either receive strings and know that they have to encode the strings locally, or bytes and know that they don't have to.

Right up near the top of Zen we get:

  >>> import this
  The Zen of Python, by Tim Peters

  Beautiful is better than ugly.
  Explicit is better than implicit.

The Python 2-style implicit conversions were awfully convenient 15 years ago when most things were ASCII. They're absolutely painful now that we've moved past the expectation that strings are arrays of single bytes.

It may just be me, but I tend to assume all strings are UTF-8.

The problem is that Python Unicode strings are indexable with integers. They support random access. Each element contains one character, not one byte. The implementation chosen for Python 3 is to handle them as arrays of 1, 2, or 4-byte items, automatically expanding the characters based on the widest character in the string. In Python 2, the options were to build the environment for 2 byte or 4 byte Unicode characters.

This could have been done differently. I would have been tempted to store strings as UTF-8 in Python 3, but hide this from the user. Most of the string functions, including regular expressions, don't really random access a string; they're sequential and forward only. They could work on UTF-8. The ones that return a string index should return an opaque type (say, "StringPos") which has a reference to the string and a position in it. The string functions should accept that type as an index.

If someone applies "int()" to a string index, or performs arithmetic on it, or indexes a string with an actual integer, then it's necessary to scan the entire UTF-8 string and create an index of where each Unicode character starts. This should be a rare event. Especially if there's support for adding or subtracting small integers from a StringPos by walking the string forwards or backwards, rather than building the full index.

Examples:

   s = "This is a test" # Unicode string, internal represention UTF-8
   i = s.find("is")     # returns a StringPos, not an int.
                        # no need to index the string for any of these
   s1 = s[i]            # StringPos selects the char found by find
   s2 = s[i:]           # No need to build an index for this
   s3 = s[i+1:]         # Adding a small int just walks the string
                        # these force generation of an index 
                        # the first operation like this on a string is expensive
                        # comparable to a UTF-8 to rune array conversion
   j = int(i)           # converts a StringPos to an int 
   s4 = s[i*2]          # again, expensive

In practice, those expensive operations are rare.

This would cut Python memory consumption way down when processing mostly-ASCII Unicode data. It would also make it more efficient to input and output UTF-8, which is usually what you want.

I recognize your name as a rustacean, and since I'm someone who reluctantly had to add some unsafe code to extract a substring due to graphemes being unstable, I'm surprised to see you take shots at the Python model.

(Granted, I'm still learning, so maybe I can be as efficient without relying on unsafe, but still...)

I'm a Rust user; I'm not involved with its development. I had dinner with some of the main developers once; that's about it.

I'd recommend using this crate to work with grapheme clusters in Rust: https://crates.io/crates/unicode-segmentation

... Due to graphemes being unstable? What do you mean?

Then I think you'd be much happier with Python 3 as it's explicitly clear when you're transforming between str and bytes objects.

I think you miss the point - if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever. By rubbing your nose in the difference between str and bytes, Python 3 complicates your life. On the flip side it makes the complicated case more tractable.

if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever

This is incorrect for anything other than an ASCII-only world. In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character.

By rubbing your nose in the difference between str and bytes, Python 3 complicates your life.

Having had a lot of experience with it, I would instead say Python 3 makes you think about things and find bugs at the appropriate time -- up-front during the development process -- rather than at an inappropriate time, like 3AM on a Sunday when your pager goes off because you trusted the "helpful" but wrong way of doing things.

> In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character.

Only if people writing code are entirely oblivious to what UTF-8 is. The most basic feature of UTF-8 is that it's a variable-length encoding scheme. If the main feature is it's variable-length, why would any programmer assume the length doesn't vary?

If the main feature is it's variable-length, why would any programmer assume the length doesn't vary?

Because UTF-8 is the Unicode encoding that looks just like ASCII, so long as the characters stay within the set supported by ASCII, so it lets them say they're doing "Unicode" while really just writing code to handle ASCII.

I am a Dane - we have three letters in our alphabet that are not represented in ascii (æ, ø and å) so let me assure you I don't except text to be in ascii, but I do expect it to be in UTF-8 and will consider it a bug if it isn't.

Bu åäöæø are still in the < 255 range so they only occupy one byte, as any other "normal" ascii character.

But they are different animals. When you read from a socket, you get bytes. Are they text or are they a JPG? Py2 says "they're text holding the contents of a JPG." Py3 says "I don't know. It's your data, you tell me."

Which works great up until the point that some existing data that's supposed to be text, like a filename, or a file in a text-based format, or string fields in a file in a binary format, et cetera, turns out to be invalid UTF-8. Maybe because it was written in the wrong encoding, or because some binary data has been stuffed in (e.g. with heredoc quoting), or due to corruption, or even maliciously as an obfuscation measure. And Py3 turns up its nose and rejects it, probably by throwing an exception and killing the script - while programs written in other languages, including Py2 (if the unicode type is avoided), 'just work'. For most strings there's no actual need to analyze them as text or look at character properties or do anything other than move them from one place to another, possibly interpolated with other strings, and in that case byte strings work fine, including with non-English text if everything standardizes on UTF-8. Therefore they should be the default, and explicit Unicode processing the exception, like Py2. The only situation where explicit decoding improves things is if you have to deal with legacy 8-bit encodings, which aren't all that common these days, and in that case you probably should be doing complicated things anyway like parsing any applicable headers, autodetecting encoding if that doesn't work, giving the user an override option, etc., which is work that types can't save you from.

Using byte strings also doesn't 'just work' if the original format is a Unicode encoding other than UTF-8; e.g. Windows filenames are UTF-16 and can't be converted to UTF-8 without risking errors caused by unpaired surrogates. But that's what WTF-8 is for (or rather, what it should be for, disclaimers in the spec notwithstanding)...

> Which works great up until the point that some existing data that's supposed to be text, like a filename, or a file in a text-based format, or string fields in a file in a binary format, et cetera, turns out to be invalid UTF-8. Maybe because it was written in the wrong encoding, or because some binary data has been stuffed in (e.g. with heredoc quoting), or due to corruption, or even maliciously as an obfuscation measure. And Py3 turns up its nose and rejects it, probably by throwing an exception and killing the script - while programs written in other languages, including Py2 (if the unicode type is avoided), 'just work'.

I'm tired of people spreading FUD about it. they either don't understand what they are doing (and do it incorrectly) or are repeating what other people (who did things incorrectly) said:

    » echo $LANG
    en_US.UTF-8
    » echo $LC_CTYPE
    en_US.UTF-8
    » echo 'works' > $'\377'$'\377'$'\377'
    » ls
    ???
    » cat $'\377'$'\377'$'\377'
    works
    » python3
    Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> a = b'\377\377\377'
    >>> a
    b'\xff\xff\xff'
    >>> a.decode()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
    >>> b = open(a)
    >>> b.read()
    'works\n'
    >>>

Accessing files with invalid utf-8 characters works just fine. Being explicit and clearly distinguishing between a string and bytes allows python to know when the filename is encoded using utf-8 (or in fact any other encoding you have defined in your environment) or bytes. Actually python2 struggles here, because it treats everything as bytes and can't tell what is used. Yes, legacy python probably won't crash, but it will spew garbage.

Python3 on the other side is strict about to point errors in your code. If python3 code will crash it's either bug in python or (most likely) bug in your code and you should fix it.

I (and I'm sure a lot of people) prefer program to crash, pointing where the error is, than silently corrupt data once in a while, and when you notice the issue spending days or months figuring out where the corruption is happening.

Okay, but where would that filename come from in a real script? If it was passed in argv, then apparently Py3 gives you a string which is invalid Unicode (contains unpaired surrogates), and explodes if you try to print it or do anything else that implicitly encodes it without the special 'surrogateescape' setting. It's really dangerous to pass such a string around. But at least you can open it, so it's not completely broken. Oh, and that means that two different strings can encode to the same path, which... is probably not usually a problem...

On the other hand, if you read a list of filenames from a file, you'd better remember to either use binary or explicitly choose surrogateescape decoding. Which is a special case of "file in a text-based format" from my original post.

I don't know what you mean by "corruption".

What I meant by corruption is the program might be spewing garbage, instead of simply crashing pointing where encoding problem is.

As for your other point. You're right, but there's no easy way resolving this. Either treat everything as unicode and remember it's binary on linux, or have linux working and everything else broken.

In that scenario you probably should use .encode('utf-8', 'surrogateescape') as soon as you can and work on bytes, so no need of doing the above.

Also, if you for example call os.listdir() with argument in bytes it'll return entries in bytes:

    Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os
    >>> os.listdir(b'.')
    [b'\xff\xff\xff', b'test.py']

But IMO, things still could be better. I would love if there was matching sys.setfiliesystemencoding() to sys.getfilesystemencoding(). In scenarios when you are not sure of encoding you could fall back to iso-8859-1 (latin-1) which has 1:1 mapping for bytes.

Currently looks like you would have to use:

    filename.encode(sys.getfilesystemencoding(), 'surrogateescape').decode('iso-8859-1')

> if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever.

Let me guess: you don't work on anything that has any internationalization ever.

Make a list of the things that work vs. the things that don't. Works: input, output, copying, formatting, numeric conversion. Doesn't: slicing, character counting. Probably doesn't: regular expressions, case conversion. Remember that all this is premised on all I/O being done in UTF-8.

So?

Even if we accept your myopic list of features that work and believe your statement that "most operations can be performed on bytes", that doesn't make the language usable for internationalized content. Trading having to call encode and decode, which fail with clear, easy-to-fix errors, for a myriad of subtle, silent encoding failures is not a good trade. Which you would be abundantly clear to you if you had internationalized anything ever.

During my last job hunt I kept getting people doing the "test for palindrome" phone-screen question.

After a while I started creatively expressing my dislike of phone screens by showing them an implementation that hints at some of the complexity lurking in this allegedly-simple task (i.e., you need to understand Unicode classes and normalization in order to write a palindrome tester).

That's totally a question I'd ask, and I don't understand why you don't like it. The reason I'd ask it is not because it's a simple task, it's because it's not a simple task: what I'm looking for is that you aren't going to over-confident-ly throw naive solutions at hard problems. That will cause bugs and have to be cleaned up.

I think if you showed me an implementation that hinted at some of the complexity lurking in the task, that would be a big point in your favor.

I think it depends on the interviewer and the purpose of asking the question. For a phone screen I'd expect it to be a variant of FizzBuzz, designed to weed out the people that aren't worth wasting time on. Coming up with a complex solution, when the interviewer was looking for something simple, is probably grounds for a fail - even if the complexity is justified.

Yeah, a lot of places use it as an initial phone-screen "can you code" question.

I did bust out a version on one call which did detection of combining classes, normalized into composed form before checking the reverse, etc., and the interviewer seemed kinda lost as I explained it.

Ended up not working for them, but for different reasons.

> Works .. Doesn't

I question some details in your lists but imagine you'd agree that the Works vs Doesn't split approximates to whole string level vs sub-string/character level operations.

There is nothing more mind-boggling than using mixed sources of ascii and unicode in Python2.

Except Python2 is the POSIX standard on how to handle this and I prefer retro OSs without a bytes/string distinction.

Python3 is the Windows way of handling strings so the change was pure churn. Both are valid, but Linux is "pretty popular" right? So why break everyone's code just to churn the pot.

If it was good for Kernigan and Richie, good for Joe Ossanna, good for Robert Pike well it's good enough for me.

The problem is it might be good enough for Kernighan and Ritchie, but it's not good enough for Chávez and Çelik (or, well, most of the world). The A in ASCII stands for "American" -- and it gives a good indication of where you can look to find a whole lot of folks who ASCII isn't good enough for.

Actually I'd have naively thought it meant America as in "The Americans" including South America. It's no wonder the IANA want's to change it into US-ASCII (or USA-SCII).

Python2 supports unicode. Chávez and Çelik are covered.

I'll take Python2's POSIX text model over Microsoft's default unicode strings every time because I prefer Linux. At this point in time see no reason to cater to a shrinking server platform.

Not to mention they didn't even get unicode right with Python3. Google did with Go (assume UTF8).

Python2 supports Unicode, but it requires careful thought and deliberation to do it right - and at the same time makes it very easy to do it wrong (by assuming all bytes are strings etc). As a result, when such code is written by people with little awareness of the world beyond ASCII or Latin-1, it tends to be buggy with locales that need more than that.

And yes, it's not really a Python issue, it's a more generic issue with treating bytearrays as strings (what you refer to as POSIX text model). Until UTF-8 became the standard encoding everywhere, there were many Linux apps that couldn't handle non-Latin1 encodings properly, either. It was also the case on Windows in 9x days, for all the same reasons.

> Python2 supports unicode. Chávez and Çelik are covered.

Most python2 code doesn't. It has a nice property of exploding spectacularly when the string you are writing out to console or file contains Chávez or Çelik.

> Most python2 code doesn't.

Citation needed. I can't say I ever had practical issues with not being able to handle unicode on Python if I needed. No matter which library. Please give practical examples of where you cannot deal with unicode on Python 2.

just plain

    for line in yourfile:
        print line

with yourfile containing non-ascii characters and running it in windows console.

That sounds like a limitation of the Windows console, not the programming language.

It's a limitation of how the programming language interacts with the Windows console, and it's a mistake. If you are writing potentially-garbled strings directly to the console, there is already no guarantee that the output will properly reflect the string or e.g. copying and pasting will work correctly, because the string could include backspace characters, or a ton of newlines. Therefore it doesn't make sense to do anything other than make a 'best effort' attempt when rendering strings to the console, including with Unicode handling. If it works, great, but if the string is invalid Unicode, that's only one of multiple problematic cases, so what exactly is throwing an exception supposed to accomplish? Python should use replacement characters by default in this case.

Stop using the Windows console. Or fix it. More rational than a breaking upgrade to Python 3.

Fixing the Windows console rather than upgrading to Python 3 is the only sensible choice.

</s>

It is, and Java has similar problems on Windows. The Windows console can be switched to Unicode, but by default, it emulates the IBM 5150 PC character set, circa 1980. It's not even ISO LATIN-1.

Works fine on Linux.

Windows console shpport is broken on python 3 too on many versions. Click the library can print unicode to the console on windows however. No python 3 needed.

> Citation needed.

I'm sorry to say, but your code for example. While I love your click library it has many small issues. Which annoy the heck out of me.

I would submit a patch but this is not a simple change and requires a bit of effort. Perhaps I'll find some time to work on it, but I'm afraid it might possibly break compatibility.

Can you talk specifics about what does not work? Because if there is an actual unicode problem in click i want to fix it.

Would have to go over the rest of code, but the code in ClickException is broken:

    class ClickException(Exception):
        """An exception that Click can handle and show to the user."""

        #: The exit code for this exception
        exit_code = 1

        def __init__(self, message):
            ctor_msg = message
            if PY2:
                if ctor_msg is not None:
                    ctor_msg = ctor_msg.encode('utf-8')
            Exception.__init__(self, ctor_msg)
            self.message = message

        def format_message(self):
            return self.message

        def __unicode__(self):
            return self.message

        def __str__(self):
            return self.message.encode('utf-8')

        def show(self, file=None):
            if file is None:
                file = get_text_stderr()
            echo('Error: %s' % self.format_message(), file=file)

This code is arguably wrong. I think I understand what you were trying to do, but I'm not certain. There is a possibility to use it correctly in python 2 though, but many people might be not aware of it.

In python 3 you pass the message as is (it might cause another issue, but about that later).

In python 2 you immediately encode the passed variable using utf-8. This means that you're expecting argument to be of unicode type, but at the same time you're discouraging users to use unicode_literals, and in most situations users will pass a regular string.

In python 3 __str__ is trying to convert text to bytes, while the message would already be text. This most of the times will look correct, but it might spew garbage when there are non ascii characters.

Here's corrected code (did not test it though):

    if PY2:
      unicode = str

    class ClickException(Exception):
        """An exception that Click can handle and show to the user."""

        #: The exit code for this exception
        exit_code = 1

        def __init__(self, message):
            if PY2 and isinstance(self, unicode):
                message = message.encode()
            Exception.__init__(self, message)
            self.message = message

        def format_message(self):
            return self.message

        def __unicode__(self):
            return str(self).decode()

        def __str__(self):
            return str(self.message)

        def show(self, file=None):
            if file is None:
                file = get_text_stderr()
            echo('Error: %s' % self.format_message(), file=file)

There are few other things that gives headaches (not necessarily python 2 only).

In python 3 for example click refuses to run if LANG and LC_CTYPE are not defined. Why doesn't it simply do what all other applications are doing and simply fall back to latin-1 (ISO-8859-1) instead of printing the error.

Also, another issue (and my above code is also is affected by this). Click is checking for environmental variables before continuing, yet encode and decode have hardcoded utf-8. It probably should use whatever locale.getdefaultlocale() returns.

Yes, it will explode if you try to write a Unicode string to something that doesn't have UTF-8 encoding by default. But it won't explode if you write byte strings, or if your default encoding is UTF-8.

Windows console is that case which doesn't default to UTF-8. At the time when I was working with it, py2 defaulted to ascii.

For some locales Windows has additional bonus, the Ansi (GUI) encoding is different than OEM (console) and it was heroic undertaking to make your program work correctly for both.

Python3 works out of the box.

I can sympathize, Windows is my environment too. But I'm not blind to other environments where some simplifying assumptions can be made.

For true insanity try adding IDLE to the mix. IDLE under Windows in Python 2 will let you output UTF-8, but won't accept it as input. Python 3 is more consistent.

Windows doesn't actually support Unicode, not really (for backwards compatibility reasons): it's just UCS-2 at the OS level, and hence you can't represent OS level strings as UTF-8 without data loss, because in practice you need to handle lone UTF-16 surrogate code units. This is why WTF-8 exists.

https://simonsapin.github.io/wtf-8/

I had thought you were being sarcastic in your grandparent post.

> So why break everyone's code

Dunno about you, but the amount of encoding/decoding errors in Py2 broke my little code multiple times in the past. "oh yeah I'll just fetch this html page"... all good, until that page contains non-ASCII and then BOOM.

With Py3 you're forced to think about this sort of issues right away and with minimal fuss you're sorted for life.

There isn't a "Windows" way of handling strings. You can handle strings however you want, just make sure that you pass the right data to syscalls. Windows APIs take UTF-16 or some other code page. Linux takes bytes. OS X takes UTF-8. There is no greatest common denominator. But if you use Unicode, then at least there's a well-defined way to write the syscall wrappers on all platforms.

And it's not like Python 3 is forcing you to give up bytes. You can still os.listdir(b'/some/dir') and you'll get a list of bytes objects back. It's just that that's not default, since using Unicode by default simply makes more sense on a cross-platform system.

actually the win api is broken when it comes to unicode. there is no real unicode in windows and thats a huge problem.

Handling UTF-8 like on Linux is definitly akward in the Win32 API. And actually that only was caused by the fact of backwards compability.

Actually it would be huge if they would force users to unicode more. But I doubt that will ever happen. We maybe see Windows Active Directory on Linux before this will happen.

It's not the API that's broken, that uses UTF-16 consistently which qualifies as Unicode by any definition.

The problem really only lies in file and console encoding. If they'd simply give you the option between legacy code pages and UTF-8, and made UTF-8 the default, life would be grand.

Yep! In my earlier days as a programmer, I wondered why all win32 API functions have both the Ansi and Unicode versions? Isn't it a messy? At the end I realized it's a honorable effort to keep old Windows programs compatible.

On the other hand, few years ago, when Delphi the RAD development tool, introduced their new Unicode version, completely breaking backward-compatibility, by changing the literal string type from "Ansi string" to "Unicode String". I know many third party libraries were abandoned and many projects were prevented to upgrade. It's also a disaster to me...

> There are some exceptions, of course.

Those exceptions are often bugs in the logic of handling text. Python 2 allows you to sweep that under the rug.

> Me, I wait for Python 5 (=2+3) which will be able to import Python 2 modules, so that you can gradually convert your code to the newer version.

Thank you for saying this! For those interested in this direction, there was a discussion about it on HN a couple of months ago:

https://news.ycombinator.com/item?id=10823406

Proof-of-concept examples embedding Python interpreters within themselves:

https://gist.github.com/dutc/eba9b2f7980f400f6287

https://gist.github.com/dutc/2866d969d5e9209d501a

Cython can probably compile your existing codebase such that it's compatible with both 2 and 3. Then you can write your new code in Python 3, while importing the old code as a C extension.

You can also do this piece-by-piece, if for some reason your code doesn't compile with Cython right away.

The compilation process also makes it much easier to identify bytes/unicode bugs, because you can choose to declare types.

That sounds like an interesting approach. Have you tried it with a real-world project? I'm curious if the approach has been proven.

> can't just take 2 months off to move all of it to Python 3

Welcome to the modern COBOL then. Have fun maintaining that and running only old libs cause everyone else in the world has passed you by.

> I'm just waiting until a consensus emerges.

Wake up, it's done emerged. Almost no libs aren't avail on 3 now.

> Almost no libs aren't avail on 3 now.

What an awkward way to say "most libs work on Python 3 now"... Why is that? I suspect it's because saying it clearly doesn't deliver the message you want.

It should be noted that the quantity of Python 2 code keeps on growing

According to this article Python2 support is shrinking.

https://blogs.msdn.microsoft.com/pythonengineering/2016/03/0...

Both claims are correct, as is common with statistics.

If you write `print "hello"`, you just increased the QUANTITY OF PYTHON 2 CODE. If there is one guy left in the world who occasionally writes some Python 2--and even if he's the only one left who does--the quantity of Python 2 code will, in some sense, "keep on growing", regardless of what happens to Python 3.

The article you cite refers to PERCENTAGE OF LIBRARIES that support Python 2 or 3, either one or the other exclusively or both, and it demonstrates a decrease in libraries that only support Python 2 and an increase in libraries that only support Python 3. This is presumably correct, too, but it doesn't suggest that nobody writes any Python 2 code anymore.

> Uglyfying my code with the "six" module it's also not a solution, since when I'll move, I won't care about Python 2 anymore.

I've upgraded a lot of my code with just future imports and it is just a tiny step from there, unless you're doing a lot of character encoding. Leave those projects behind, but moving most other projects is easier than I expected.

> I have tens of thousands of lines of Python 2 code in a big system. I can't just take 2 months off to move all of it to Python 3. Moving it in pieces is also not really possible, since there are many inter-dependencies.

If you cant evolve a system incrementally because of excessive coupling, that's a problem, sure, but not a problem of either the platform you are on or the one you might be thinking about moving to, unless the former forced a tightly-coupled architecture on you.

We've solved this problem by breaking our architecture up to be more distributed and introducing Python 3 one small service at a time.

Same here. It's also been a great opportunity to step back and reevaluate some early-startup engineering decisions that've made the new service codebase fun to work with. Rather than having a single backend that has to handle all possible situations ("this one thing we're talking to speaks XML..."), we can only deploy the stuff actually needed for that one service ("I think we can safely assume all-JSON here"). I highly recommend this approach.

Can you provide few irreconcilable issues that stops you from using python 3?

> Moving it in pieces is also not really possible, since there are many inter-dependencies.

I guess something is wrong with your architecture in first place, so you can't separate domains or layers easily.

I've used this module for upgrading a python2-only lib : https://pypi.python.org/pypi/future . Basically it allow you to write Python 3 code that will run on Python 2 interpreter (bar from using advanced features like coroutine or matrix mult operator).

Once you decide not to support Python2 anymore, just drop the imports. They also have a Py2/3 compat cheet sheet : http://python-future.org/compatible_idioms.pdf

It is pretty obvious there is a problem when your quick and easy cheat sheet is a 33 page tomb. If only Guido and company would acknowledge the problem and fix it instead of putting their fingers in their ears pretending Python 2 will cease to exist in 2020.

They acknowledge the problem, and their solution is to stop actively aiding and abetting the people who are continuing to make it worse by writing python 2 code.

Wow, what a simplification.

What about us who have and want to use massive libraries of existing Python 2 code?

"Just upgrade your libraries!" is not realistic for everyone, especially those who have to justify a mostly idealogical conversion to a similar language, and spend real time and money, outside of actually developing the end product, to do it.

Four more years of official support is a long time.

Someone suggested Cython to compile python 2 code as modules and import to python 3. Did not try it, but that sounds plausible.

The biggest obstacle people seem to complain about (besides print, which is trivial IMO) is unicode. This is generally also the most painful part of making python 2 code work on python 3.

The issue with it is that from unicode perspective your python 2 code is inherently broken. There's no automated tool that can fix it, and there's no way python 5 will be able to run python 2 code correctly unless unicode support would be removed. You have to manually tell python which string is made of characters and which one was made out of bytes.

You can automatically remove six from your codebase after you're Python 3.x only. So six can be a possible solution (provided you have automated tests).

Reply on news.ycombinator.com