Hacker News Reader

Comment by nodamage | Hacker News Reader

I have a Python 2.7 project that has been running smoothly for many years now and I'm having trouble finding a reason to upgrade to Python 3. The project uses the unicode type to represent all strings, and encodes/decodes as necessary (usually to UTF-8) when doing I/O. I haven't really had any of the Unicode handling problems that people seem to complain about in Python 2.

Can someone explain what benefit I would actually gain from upgrading to Python 3 if I'm already "handling Unicode properly" in Python 2? So far it still seems rather minimal at the moment, and the risk of breaking something during the upgrade process (either in my own code or in one of my dependencies) doesn't seem like it's worth the effort.

View on HN

You gain access to continued language support in 2020. New features involving strings will have less risk of bugs. The "range" function is more memory efficient. Integer division automatically floors, reducing bug risk. Dictionaries with guaranteed ordering. Thousands separator in string formatting. Bit length on integers. Combinations with replacement on itertools. New, faster I/O library and faster json. Concurrent futures module. Ability to define stable ABI for extensions. New CLI option parsing module. Dictionary-based logging configuration. Index and count on ranges. Barrier synchronization for threads. Faster sorting via internal upgrade to Timsort. Async I/O. Support for spawn and forkserver in multiprocessing. Child context in multuprocessing. Has collision cost reduced. Significantly faster startup. Type hints. Faster directory traversal. Faster regular expression parsing. Faster I/O for bytes. Faster dumps. Reduced method memory usage through better caching. Dramatically less memory usage by random. Faster string manipulation. Faster propert calls. Formatted string literals (interpolated strings). Asynchronous generators. Asynchronous comprehensions.

How much faster might your code run just by upgrading to Python 3? How much memory might you save?

> Dictionaries with guaranteed ordering.

I don't think you're suppose to depend on the ordering of dictionaries. It's an implementation detail which might get changed, although it wont actually ever be changed because people will come to depend on it.

I'm specifically referring to OrderedDict in this case, which does have guaranteed, insertion-based ordering. It was introduced in 3.1, circa 2009, via PEP 372.

https://docs.python.org/2/library/collections.html#ordereddi...

> New in version 2.7.

That's a good point. I didn't realize that it was also introduced into 2.7 at the same time.

> I don't think you're suppose to depend on the ordering of dictionaries. It's an implementation detail which might get changed, although it wont actually ever be changed because people will come to depend on it.

I came here to make the same distinction - though I will say I hope you are wrong about people coming to depend on it when OrdereredDict is still there for a reason. The docs still plainly state that Dict should be considered un-ordered and do not make mention of this implementation detail (nor should they).

You forgot functools.lru_cache. I've found in some Python 2.7 projects you can speed things up with a hacky dictionary-cache, but the proper functools.lru_cache gives better results and is far more flexible.

Like many database-backed apps performance is currently limited by I/O, and memory is limited based on the size of the data sets I load (machine learning). A faster, lower memory Python does sound nice but I'm not sure how much effect it would actually have in real life.

What py3 does is it enforces the things that py2 only suggests as far as unicode goes. That means on py2 you can go for the proper handling and if you did do everything right, then py3 migration should be almost boring.

The difference is when you didn't handle all the cases correctly. In that situation Py2 will silently do either the right, or the wrong thing and you'll never know. Py3 will likely throw an extension and tell you where the issue is.

So if you're handling lots of unicode text and think you've handled all the string correctly already - that's the reason to move. Now you'll be sure it's correct. The problem with py2 wasn't so much that it was handling anything badly - it was that the default, easy way was incorrect and just waiting to blow up in peoples' faces. (or maybe just silently corrupt something)

I'm fairly sure I'm handling strings correctly under Python 2. What I'm not sure of is the risk of something breaking if I upgrade to Python 3, especially when it comes to upgrading external dependencies. Even if it's just non-backwards compatible API changes, it's more risk to deal with.

You're fairly certain, but with Python 3 you can be completely certain. That's the magic of it.

Python 2 will allow this to sometimes work, Python 3 will make sure this never works, because there's a bytes/text mismatch. It's nice to rule out entire classes of bugs like this

    from sys import argv
    import json
    with open(argv[1], 'rb') as f:
        json.loads(f.read())

Porting is still difficult. But when we ported from Py2 to Py3, we found a couple issues like this (despite dealing with weird encodings all the time).

Exactly this. When I ported my stuff to Python 3, I fixed many encoding bugs in the process.

You can do this correctly in Python 2, it's just much harder.

Do you think you will ever have to move to Python 3?

I don't know. I will probably try to by 2020 once it gets officially EOLed. Until then I have plenty of other work to do.

Isn't that what test bases are for?

Tests will tell you something broke, you still have to go through the work of figuring out why and how to fix it.

It has always seemed to me that Python 3 was mainly a fix on the philosophy of handling strings, but that it didn't offered a clear practical advantage for programmers already handling strings with care. I don't think there is a practical reason to upgrade to Python 3 in terms of language design. The reason will be in term of survival as the community seems to be willing to follow the Python 3 movement and official support for Python 2 ends in 2020.

There are a lot of reasons to upgrade! There is so much more useful stuff in python 3!

Even if you think you would not use those features, other libraries you may use might benefit a lot from it.

A few features: async/await, lists (and others) use iterators, no var leaking in list comprehensions, super().my_method() instead of super(MyClass, self).my_method(), class MyClass: instead of class MyClass(object):, improved exception handling, required arguments, ", ".join(["etc"]* 1000)

Besides that: a much improved standard library, although that technically not is language design.

Of which the improved exception handling is my favourite.

You could do ", ".join(["etc"]* 1000) in python2 - what am I missing?

ah well, it was just a geeky way to say etc etc etc.... no python3 stuff intended to be used there. Sorry :) This one then, althoug it sort of works in python 2 as well..

", ".join(['ètç']* 1000)

Ha! Sorry about that :-)

Yes, that is my impression as well. In hindsight, it is unfortunate how much it ended up fracturing the Python ecosystem.

If everything fine with Python 2 on the current project, why bother to upgrade Django 2.0 which will break the compatibility?

That's pretty much true of any django release... you can stay on it if you so choose to. It will get security updates/bug fixes for a bit and then they will stop coming. You're free to stay where you are if you want.

It turns out that most developers have a desire to move to the next version if it's not too hard. There's still COBOL programmers out there too and that's perfectly fine.

Django has made the process as smooth as it can be. You can upgrade to python3 while maintaining your Django version. Then update to the next Django version as a separate step. It's fine to have waited until now. You can keep waiting if you want but it's getting to the point where you should really just do it. It's not so bad.

We switched and python2 --> python3 was bumpier and more work than most Django updates we've done (we've done pretty much every one since 1.0) but it was still entirely reasonable. We're much happier now.

Right, that's more or less my question. (I'm not actually using Django though.)

My feeling is that you want to be using dependencies that are actively being maintained. These probably already support py3 - or there's a newer / maintained alternative available.

What's annoying is discovering that it's a struggle to upgrade to a newer OS because you're using some old python dependency that has some C component linking to some library that you're going to spend a week getting working (and then have to continue to maintain).

It sounds like you might not have too much trouble upgrading anyway. The strings and missing libs are the places most people get caught out and it sounds like you're already handling the worst of those.

If I were you, I'd try switching to python3 and see what breaks. When I did it, it took about a day to get up and running again on a reasonably complex project (numpy, scipy etc). One of the main things I ran into was places where python3 had swapped lists to generators etc (eg, some_dict.keys()[0] no longer works).

Well, there are some new nifty things, e.g. not having to write u in front of every string or a more pleasant subprocess API or the fact that dicts in 3.6 keep their insertion order which makes debugging a bit easier. It adds up.

But the big one is that Python 2.7 will go away at some point.

"I have a Python 2.6 project that has been running smoothly, why upgrade to 2.7?"

If you don't need any of the new features and fixes, don't, that's perfectly fine.

The difference between going from 2.6 -> 2.7 and 2.7 -> 3 is enormous though.

Not if you don't need it.

I meant in terms of work needed to upgrade.

> Dictionary-based logging configuration

I believe this already exists: https://docs.python.org/2.7/library/logging.config.html#logg...

Here you go: https://eev.ee/blog/2016/07/31/python-faq-why-should-i-use-p...

Reply on news.ycombinator.com