Python 3 Features | Hacker News Reader

Python 3 Features

asmeurer.github.io

109 pointsSami_Lehtinen79 comments

Very nice and fun presentation.

Another feature not in this talk is function annotations (http://legacy.python.org/dev/peps/pep-3107/) and things they enable like optional runtime type checking (https://github.com/kislyuk/ensure#enforcing-function-annotat...).

Also, as a library developer, I wanted to highlight some facts from experience with Python 2/3 development:

- Travis CI is an invaluable resource for running your test suite on a number of Python versions at once. It has deep integration with GitHub, and together they make the development flow a lot less cumbersome.

- Writing to a subset of Python that is compatible with both 2.7 and 3.3+ is not terribly hard. Lots of resources exist out there, including Armin's porting guide, six, python-future, and eight (https://github.com/kislyuk/eight).

- Contrary to some opinions out there, Python 3 actually makes working with Unicode a lot less perilous than Python 2, mostly by having a sane encoding default (UTF-8 instead of Python 2's ascii) and not doing automatic bytes/unicode coercion.

- PyPy is awesome and just released their first non-beta Python 3 version. Please donate!

I guess I'm in the minority here of being a full time python developer and thinking 1) these are all totally unexciting for day to day python use, and 2) the only actually interesting one (async io) is covered with:

    Not going to lie to you. I still don't get this.

...which is not really a call to arms for py3 as far as I'm concerned.

(pypy support and potentially having one binary running 2.7 and 3.x at the same time is much more exciting)

I agree. I use Python3, but for none of the reasons given. Some of the advice was just flat out bad, IMO.

Like replacing

    def wat(a, b, *args)

with

    def wat(*args)
        a, b, args = *args

I'm sure (I hope) that was just a proof of concept. It's a horrible idea!

No one in the Python community is advocating the "replacement". I think it's just an example of new syntax.

Slight typo - the sample replacement should read:

  def wat(*args)
    a, b, *args = args

..But I don't see how moving required arguments from the signature to the body (or turning a TypeError into a ValueError) might be useful, or how it differs from the presentation's first advanced unpacking example.

I'm exactly in the same boat than you. Worst, I do believe that, even if the API is nice, asyncio belongs to the old past way of doing concurrency, I want actor models and/or STM with feature like the ones found in clojure.

Python core devs really miss the point that if you want people to move from one solution to another with a cost (here: breaking backward compatibility) you have to provide an high value reason to move (see the network effect). Here we only got minor (but cool) improvements, moving to python3 is just not worth it. It's like having to do a plain old boring homework imposed by a teacher with the only reason "because I told you to" (and the situation is even worst if you have to do compatible code for libs).

I'm not at all against breaking backward compatibility, you have to have a good reason and to make it worth it and, of all the (minor) improvements I see, none of them was a justification for this. Tell me things like "oh, we break backward compatibility because we want to drop the GIL and implement the erlang actors model natively in python" and I'm with you the day it's released (even if you don't provide automatic CPU balancing at the beginning).

There's also a python 2.7 backport of asyncio that works perfectly fine for me.

https://pypi.python.org/pypi/trollius

I think asyncio + "yield from" is the killer feature for Python 3.x right now. (It's covered briefly in these slides.)

Beazley gives it a very detailed treatment in this amazing YouTube video, which covers not only asyncio's internal implementation, but also explains what coroutines are and when you'd actually benefit from using them in async programming:

https://www.youtube.com/watch?v=5-qadlG7tWo

how does asyncio deal with resource management, a very hard problem whenever you invent lazy io?

Chained exceptions* is fantastic. I've found myself handling a specific set of exceptions, logging/printing various pieces of the traceback, and continuing on more times that I'd care to think about. This provides the level of context to the exception I'd have wanted.

Combined with fine grained `oserrors`, that's awesome. I have so much magic `errno` checking code over the years.

* https://asmeurer.github.io/python3-presentation/slides.html#...

> Feature 0: Matrix Multiplication

It’s not really a feature, it’s an operator that can be overloaded. That is, instead of using `mymodule.dot(a, b)`, you can use the much less readable `a @ b`.

> Feature 1: Advanced unpacking

> Refactor your functions

The given example is a bad refactoring, because it moves away the function’s self-documentation.

> It’s not really a feature, it’s an operator that can be overloaded. That is, instead of using `mymodule.dot(a, b)`, you can use the much less readable `a @ b`.

I disagree with you, and think it's definitely a feature. If you are doing scientific computing it can be much more readable to use infix overloaded operators. What is better, `np.dot(x.T(), np.dot(M, x))` or `x.T() @ M @ x`? I find the second much more readable.

I get it that operator overloading can be abused and lead to unreadable code, but in some use cases it is a godsend.

That’s true, but you already can overload other operators like __mul__ to have `x.T() * M * x`.

__mul__ has been used for elementwise multiplication in numerical python for the past 18 years or so. You can argue if it was a good choice or not, but it's a bit too entrenched to change now.

Yeah, I didn't get the refactoring either. Unless it's a jab at Perl.

Too little too late and not good enough,very little offered for splitting the user base. Python in the long run is badly positioned as

1. Moores law partytime is over. Performance matters again and the language has significant upper limits on speed.PyPy is not fast enough and further splits the user base.

2.As performance matters again concurrency matters more,concurrency in python in horribly implemented.

3.AsyncIo, What the hell am I even looking at? Gevent is vastly simpler than this.

It is a pity as it is one of the cleanest languages around and a joy to code in. But programmer productivity is more important than speed you say? We in many cases that is true but languages like go and Julia can offer both so why program in a slow language which you may have to replace later when its almost as easy to take another option and have everything?

I really don't see how PyPy is "splitting the user base" when you have to make close to no changes to your code to gain an appreciable speedup in 90% of the cases.

I see many people compare Python to Go and frankly? Go isn't nearly as nice to work with as Python (no exceptions? one among many small but noticeable annoyances). It's much nicer than C or C++ in cases where you require the raw performance but it isn't a replacement for Python.

There are also many areas where performance is literally of zero consideration (small scripts to automate stuff) and those areas are where Python is hard to beat.

I think the problem with Go coming from Python is that the paradigm shift is a lot more substantial than it appears at first blush. My experience was quite the opposite: Go is pleasant to write in, exceptionally simple, and once you fully grok (and appreciate) slices and channels, the expressiveness of the language starts to take a life of its own. I never missed exceptions much (again, because you have to completely accept Go for what it is) and the fact the language specification is so short makes it easy to understand and relatively fast to pick up. Plus it's fast.

My problem with Go has nothing to do with the language per se. Instead, my reservations fall mostly on the ecosystem. It's immature and compared to more established languages like Python, it isn't nearly as rich (after all, it isn't as old). I gather from reading the mailing list archives off and on that there's been some inertial resistance toward certain frameworks (I can't really fault anyone for this--Martini and Revel have too much "magic," for example), and in some cases, prospective users are advised to "roll your own" if it's not something in the standard library. But when it comes to certain odds and ends, it's just downright difficult at times to find something that works as well as its Python counterpart out of the box and is relatively API stable (blackfriday, the Markdown parser, comes to mind--but then I guess offloading that to the client using JS might be better these days?).

Also, the abuse of the word "idiomatic" grates on me a little. There was someone on HN a number of months back who joked (paraphrasing) "Great, we're going to see 'idiomatic' abused as much as 'pythonic' and 'pivot.'"

But, that's why I do agree with you: Sometimes performance doesn't matter quite as much as the ability to avoid "not invented here." And Python has libraries for almost everything--SQLAlchemy is probably one of Python's "killer apps." (I expect Go will progress there, one day, but for some use cases it's not quite at that point in my experience--so the answer is probably to contribute to those Go libraries/projects you find of use.)

Pypy3 + STM is coming, which should resolve your first two points. Pypy is fast enough for most, and is gradually getting faster. I don't see at all how it splits the user base.

With regards to asyncio, a lot of people would disagree with you.

Too little too late? Maybe...that's for you to decide. Yes, there are new competitors that didn't exist when the decision was made to pursue the Python 3 strategy. But Julia has nothing close to the standard library + ecosystem that Python does, and probably won't for a long time if it ever does; and while Go is coming along in that regard, there are some aspects that make it less preferable to Python, especially JITed Python with software transactional memory (i.e. no GIL). I, for one, would never want to code without exceptions if I had the choice.

> PyPy is not fast enough and further splits the user base.

There is no split?

> Gevent is vastly simpler than this.

I prefer asyncio's explicitness to gevent's monkey-patch-and-pray-it-doesn't-block semantics. Also, the asyncio approach makes it very clear when you have released the "lock" on your state--it happens every time you yield from your coroutine. The gevent approach is to hide these synchronization points, which I think is a bad policy. For what it's worth this latter objection is one of the main reasons asyncio is the way it is--GvR had been bitten before by implicit yields.

If you're needing to think carefully on what code you call might cause a switch and what won't, you're already doing it wrong. Instead of treating gevent like coroutines with less predictable ordering, treat it like threads with more predictable ordering. You know that basic operations (eg. checking if something is a certain value, or assigning a variable) won't cause a switch. So you can do two of those in a row, without thinking about race conditions. Anything beyond that...use a lock (or a queue, or any of the other provided concurrency primitives).

Well, thanks to operator overloading, comparisons or assigning a variable (to a property of an object) can cause a switch. If you really need to get a mental model of what your code is doing, or could do under sufficiently unusual circumstances, gevent doesn't really provide any guarantees beyond raw threading.

Glyph (of Twisted fame) wrote a good blog-post about it: https://glyph.twistedmatrix.com/2014/02/unyielding.html

gevent simply has the nicer API. I've never once had an issue where the monkey patching failed, or conflicted with another library. Yes, monkey patching is a dastardly and gross thing, but in practice gevent works amazingly well for 99% of use cases.

Not to mention the performance is very good, especially in the most recent version.

Couldn't agree more.

Python is essentially dead as anything more than a scripting language.

I find no reason whatsoever to move to Python 3. These "features" offer nothing of substance to me and my typical workload with Python. Finally, Unicode handling in Python 3 is a huge mess.

No thanks.

I must admit, some of these features would be very nice. I'm sick of manually dealing with kwargs just because I wanted a keyword arg AND a *args.

...but you're right, it's not worth the unicode handling.

Later: "10 awesome libraries you can't use because they haven't been ported to Python 3"

Can you cite me 10 awesome and useful libraries that haven't been ported to Python 3 and don't have reasonably equal alternatives?

I know there are some critical libraries to some niches who don't particularly care about recent versions. I know there's a fair amount of libraries that don't work on Python 3 purely because the author used print instead of print(). I know there are apps that can't port to Python 3 because they were designed with some core Python 2 functionality a decade ago (usually unicode/bytes related).

None of these matter to real world devs. User-facing apps don't matter, easy-to-port libs don't matter (fork & fix), niche libs don't matter (if your niche doesn't care about Python 3, you're either here to troll or you're not actually here).

I know it's very "in" to say that Python 3 is dead-on-arrival and all that but it's simply untrue. Anyone who uses a Python 3-based Linux distribution will understand that. Most new devs use Python 2 because they are too lazy to install Python 3 and assume their code won't work; and as soon as they hit a SyntaxError in print they complain that "Python 3 is hard".

Here are the facts: https://python3wos.appspot.com/

You see all these libraries in red? Most of them depend on each other (Twisted is a big one as well as another I forget about) and have excellent alternatives available. Good luck finding 10 that match my criteria; I can't even find one.

Often one does not need 10 reasons, 1 is enough if it's critical enough.

For my case the missing libs in py3 world are

* boto (AWS interface) * pika (RabbitMQ interface)

Yes, supervisor too, but it can run in a separate Python 2 env.

It's also about fear that next package that we need to progress faster might not support py3.

But confidence builds over time and the list of Python 3 packages will only improve over time. It will converge to one single Python 3, there is absolutely no doubt.

Pika has a working python 3 tree, I have been using it for a few weeks now.

if your niche doesn't care about Python 3, you're either here to troll or you're not actually here

Please explain. As someone who will be writing Python 2 for the foreseeable future due to niche libraries, I really don't understand what you're trying to get at.

I was in your shoes a few times. I wanted to use Python3, but sime niche library I depended on was Python2 only.

And you know what, I simply ported them. Took me a week or two, and I learned a lot. I have been using Python3 ever since.

MySQL-python is the one that prevent me from using Python 3

I highly recommend you switch to Postgres which does not have this issue with psycopg2. I know it's not always possible, but seeing as you work with Django it's a generally good idea to spend the effort.

Yup, this certainly needs work.

Myself, I was very skeptical, until one day I decided to start my new project in Python 3 and quickly realized that everything isn't that much ruined. Couple of pull requests here and there, could be worse (after reading some of the more... opinionated blog posts I expected something truly catastrophic).

But the new goodness in 3.4 is totally worth it, in my opinion.

This web application tracks the Python 3 compatibility status of Python's most popular libraries.

https://python3wos.appspot.com/

It's mostly green. If there is a red one you really need, it's probably not hard to port it.

Boto, pika, protobuf and thrift are still red, and probably nontrivial to port. It would be interesting to know if str/bytes problems are the main issue though.

What about https://github.com/GreatFruitOmsk/protobuf-py3?

I don't really see what could be non-trivial about Protocol Buffers. The encoding's simple to the extent one could probably hack a minimal working implementation over a weekend or another. That is, unless you need not the data format, but the very exact Google's library (which, to my tastes, is severely unpythonic and has an quite weird and bloated codebase - which is probably the issue with porting, but sadly is a dependency for some other libraries).

Yes, sticking with Google's library is the limitation. Without the official library, there are probably better ways forward anyway.

Somebody please port MySQLdb and make it asyncio-compatible.

The unpacking is cool, reminds a little of pattern matching destructuring binds in Erlang etc.

If Python 3 is so wonderful, why are the Python 3 developers still using Python 2 in their toolchain? It's because it would reportedly take 1 man year to port Mercurial to Python 3.

If an organisation has lots of legacy Python code, it'll cost them a lot of effort to re-write it for Python 3, and for what? Programmers might be maybe 1% more productive in the new language than the old one. For most organisations, this would not make business sense.

If Go is so awesome, why are Go developers still using C in their toolchain?

That's a terrible argument. Of course it takes effort to port existing application and tools. In some cases, it might be worth it and in some cases, it might not be. Python 2.7 will always be available to run such tools and applications.

This is very different from the case of library developers that should port their libraries to Python 3 (or support both Python 2 and 3, this is not difficult) so that it's accessible to more code.

This isn't an argument at all when it comes to writing new stuff in Python 3.

That argument makes no sense.

Mercurial isn't part of Python, and is essentially a black box tool. We use mercurial for a C++ project at work. Does that make it a Python and C++ project? Obviously not.

You might as well argue that if Python3 were so great the devs wouldn't be using C and C++ in their toolchain, which is what most of the underlying operating system and tools are written in.

And in that case, the argument would apply to almost every language and implementation out there.

Because the auto-unicode of system interfaces is a PITA if you want to deal with opaque data which is either binary or an unknown encoding?

Because the incompatibility of unicode and bytes (b'x' != 'x') and the "everything must be unicode" dogma of the ecosystem make it impossible to use many libraries that previously worked just fine ignoring any non-ascii characters?

If nothing else, because it's really really annoying to have to write "b" in front of every string literal just so I'm actually dealing with strings, not text?

Unicode-by-default may be dandy for a web developer, but for using python for system applications and glue logic it just makes things difficult.

I will never move to python3 for as long as they've broken strings.

> It's because it would reportedly take 1 man year to port Mercurial to Python 3.

Source?

> If an organisation has lots of legacy Python code, it'll cost them a lot of effort to re-write it for Python 3, and for what?

No need to port code that doesn't get a lot of attention. But if you want new features of Python, then you just have to bite the bullet. Organisations upgrade hardware all the time. They upgrade their OS somewhat regularly (on the order of years, but still). You get to make a choice - stay current and supported, or don't.

As for me, I will be (and have been) writing new applications with python 3. But I still maintain and support a number of 2.7 apps which will probably never be upgraded.

>> It's because it would reportedly take 1 man year to port Mercurial to Python 3.

> Source?

Python 3.x has proven notoriously difficult to support, due to our pervasive dependence on a byte-based encoding strategy and string manipulation. We have invested a substantial amount of time into porting at this point and we estimate at least a person-year of core developer time to make a full 3.x-compatible version.

-- http://mercurial.selenic.com/wiki/SupportedPythonVersions

- They don't mention in the presentation that you should port your current projects to 3.x Use it for your new projects.

- It would take a lot of time for Mercurial team because they work with files and have to support a lot of cases, encodings and stuff. Python 3 does a lot for the developer now.

- Mercurial is not the best example because you aren't supposed to use their internal API anyway. It's their choice to port it to 3 or not.

> If Python 3 is so wonderful, why are the Python 3 developers still using Python 2 in their toolchain? It's because it would reportedly take 1 man year to port Mercurial to Python 3.

Why would that have anything to do with it? We didn't write it and don't have any input into Mercurial, and it works fine for what it's used for right now. It's a source control tool, not a library we use in any way.

If we did the subversion switch today, it'd be to git, which again has nothing to do with Python versions (obviously).

What are some of the indicators that core devs have more interest in git than they did back then?

Conversations at the language summit, mostly. A lot more of us use git more often than we do hg these days as well.

Wasn't aware. Good to know, thanks.

This is why web companies need to use service-based infrastructures. : )

It allows you to be agile about technologies without porting the world.

How long would it take to port Mercurial to Go?

Code which is costly to port won't be ported to anything, it will just sit there in its current state unless someone decides to port it.

At this point, shouldn't organizations have a Python 3 plan? Legacy support ends in 5 years. Shouldn't new apps be written in 3 when possible?

> Legacy support ends in 5 years.

Really? Support for Python 2.7 was going to end in 2015, but they recently extended it to 2020.

What's the betting that in 2019 they won't extend it to 2025, followed shortly after by quietly dropping python 3 (or someone forking 2.7 to produce a compatible 2.8).

> Shouldn't new apps be written in 3 when possible?

There are no compelling reasons (for me at least) to change to what is in fact a different language which happens to use the same name.

When the Python 3 people eat their own dogfood by using a Python 3 toolchain, I might reconsider.

2014 + 5 years? Should your company fire you if you're wrong about an extension to 2025? You're going to cost them a lot of technical debt.

I always wondered how we got to the Y2K problem. That was a pretty hard deadline and people knew it was coming for years. How do people get away with creating problems like this?

LRU cache for py2 - http://code.activestate.com/recipes/498245/

Surprised no mention of concurrent futures, which also has a py2 backport - https://pypi.python.org/pypi/futures

What are some of the best tools for creating such slides? I see that this one uses remark.js. What else are people using?

Well, I can tell you the choice here wasn't good because I can't use my mousewheel to scroll through it. Which is awful.

Impress.js is far much pretty, better and a little heavier, too.

Good move. Infact I'll suggest that the devs have some courage and make it a new language entirely. Trying to keep python3 in an awkward backward compatibility is harming it. And remove the GIL limitation at all costs.

Getting the first and last line of a file with unpacking is pretty slick. Respect.

Could somebody explain what all the hype is about generators? Why is this better practice? Optimization, readability, what's up?

It's about RAM usage. I strongly encourage you to read this: https://wiki.python.org/moin/Generators.

I see, great! That is exactly what I was looking for. Thanks!

Can't get past the first slide on phone (IE 11, WP 8.1). That's pretty lame.

Edit: I guess that's why the PDF link is there.

The trouble is for a good number of projects I need 2.7 because of libraries being out of date.

Consider this Python 3 code extract:

    HCK = 3

Now, does that identifier consist of three ascii letters, or 3 Cyrillic letters that happen to look the same, or some combination of the two? It's impossible to tell.

In Python 3, identifiers don't identify!

Consider any bad code. What does it do?

Supporting more than ascii in code is a wonderful, wonderful feature for some countries and opens up Python to people for whom the latin alphabet is as familiar as the [pick a non-latin alphabet you don't understand] is to you.

Would I use it? No. Would I work on code that used it? Not in three lifetimes. Do I like that it's there? Hell yes.

As the slides were showing, I can write shutil.rmtree("/"). Does that mean rmtree should prevent me from passing "/"?

> Supporting more than ascii in code is a wonderful, wonderful feature for some countries and opens up Python to people for whom the latin alphabet is as familiar as the [pick a non-latin alphabet you don't understand] is to you.

No it doesn't, because reserved words and all the identifiers in the standard library are still in Latin. It is simply not possibly to write Python code without knowing the Latin alphabet.

There is a difference between learning the 33 keywords and learning and dealing with an entire foreign alphabet.

And you would be surprised what "development translators" can come up with. You can write wrapper libraries for, heck, the entire stdlib. Yes, it's stupid; but if you are looking for devs in a country where English is less spoken than French on Saturn, it's actually useful.

> There is a difference between learning the 33 keywords and learning and dealing with an entire foreign alphabet.

It's 52 letters, that're used everywhere in computing. Are there really large numbers of Python programmers who have difficulty with it? I doubt it.

> but if you are looking for devs in a country where English is less spoken than French on Saturn, it's actually useful.

Have a look at the most-downloaded Python packages at https://python3wos.appspot.com/ . How many of them have documentation in languages other than English?

> Are there really large numbers of Python programmers who have difficulty with it? I doubt it.

Take a guess why that is, please. I'm not going to do a crappy analogy here, but you can't expect "large numbes of people" to be doing something they have inherent difficulty doing.

There is a difference between learning the 33 keywords and learning and dealing with an entire foreign alphabet.

Perhaps, but notice that in order to merely type in those keywords, you need to use pretty much the entire basic Latin alphabet (except for j, q, v and z) [1]. So I think it's fair to ask how big of a difference it really is.

[1] https://docs.python.org/3.0/reference/lexical_analysis.html

There are countless examples of where allowing non-ASCII identifiers makes code more readable in non-English languages. For example a Brazilian coder might write

país = "Brasil"

Without the accent, pais means fathers rather than country.

Imagine how constraining it would be to code in a language that disallowed vowels in identifiers (perhaps because it had been designed by Hebrew speakers). That's how most of the world experiences Python 2.

Sure: your point feels right logically and politically, but in fact it doesn't quite work that way: for Latin script-based languages like Portuguese, non-ASCII letters are extensions (diacritics or liaisons) of the basic Latin alphabet that are often left out in informal writing (eg by grade students, when texting/chatting, or by "uneducated" speakers) when the context is clear and a romanized phonetic approximation is available. As a result, native speakers are well used to homonymy in daily use, even though it might appear quite confusing to a non-native.

So, in practice, ambiguities like the one you point out are more likely than not to be as constraining to Latin script-based programmers as common English homographs like "bank" do.

As a Portuguese coder myself, I shudder seeing that code. You know how would that be written at my company?

  country = "Brasil"

Code in languages other than English is just painful.

Reply on news.ycombinator.com