What’s New In Python 3.6 | Hacker News Reader

What’s New In Python 3.6

docs.python.org

273 pointsorf126 comments

First of all, congratulation and a huge thank you to the core python development team.

I know some of them have expressed frustration (or more negative feelings) at the way some people have been reacting to their work. As the Python ecosystem is progressively moving to Python 3 (according to some stats, the tipping point should be reached in a few months), I hope that this trend of excellent Python 3 releases will continue.

Just as an anecdote: I haven't heard _any_ criticism about Python 3 outside of the Hacker News community. In Germany, everyone loves it, if only for the improved Unicode handling.

Given that the HN community might represent a decent chunk of the overall Python-using community, I wouldn't interpret that as HN being whiners for the sake of whining.

For starters, Armin Ronacher has written thorough essays on his dislikes. He comments on HN but his influence is obviously much bigger as a Python developer (Flask, etc).

In academia/science, maybe people aren't ripping on 3.x explicitly, But that's because they have the ability to control the work environment and stick with 2.x. They don't rip on 3.x because it's non-existent as far as they are concerned.

Example: http://people.cs.umass.edu/~sheldon/teaching/cs335/

> Programming assignments will use Python, NumPy, and SciPy. The required Python environment is the Anaconda 4.1.1 distribution of Python 2.7. We will not help grade or debug work unless you are working in this environment.

> I haven't heard _any_ criticism about Python 3 outside of the Hacker News community.

That only means you don't participate in any other online community that discusses Python 3 besides Hacker News.

For instance, the Python2 Vs Python3 migration issues are widely known even for those who never written a single LoC in Python.

     Note Prerelease users should be aware that this document is currently in draft form. It will be updated substantially as Python 3.6 moves towards release, so it’s worth checking back even after reading earlier versions.

Although this document is dated today, but it bears the above note right below editors.

Also according to PEP 464's Python 3.6 release schedule, Python 3.6.0 final was scheduled on 2016-12-16.

So the title should be changed to "Python 3.6 release candidate 1 released"

The dict shrinkage(25% less!) in memory is pretty sweet! I'm not sure how many of you abuse dicts to hold [just about] everything, but this is a big boost for those of us that do.

I've been trying to break this habit and get into using namedtuples, but the problem is I still need to look up the API every time I need to use them.

Besides immutability, what are the advantages of using namedtuples instead of dicts?

- A clear single point of truth describing your data.

- Probably more memory efficient.

We can test memory usage in CPython (3.5)!

  >>> import collections
  >>> import sys
  >>> class A:
  ...     def __init__(self, x, y):
  ...         self.x = x
  ...         self.y = y
  ... 
  >>> class B:
  ...     __slots__ = ('x', 'y')
  ...     def __init__(self, x, y):
  ...         self.x = x
  ...         self.y = y
  ... 
  >>> C = collections.namedtuple('C', ('x', 'y'))
  >>> a = A(1, 1)
  >>> b = B(1, 1)
  >>> c = C(1, 1)
  >>> d = {'x': 1, 'y': 1}
  >>> sys.getsizeof(a) + sys.getsizeof(a.__dict__) # Plain object
  344
  >>> sys.getsizeof(b) + sys.getsizeof(b.__slots__) # Plain object with slots
  120
  >>> sys.getsizeof(c) # Named Tuple
  64
  >>> sys.getsizeof(d) # Dict
  288

EDIT: Fixed a slew of measurement issues

EDIT2: Included size of __slots__ tuple

If you're using Python 3.5, use tracemalloc, it is much nicer for this sort of thing.

https://docs.python.org/3/library/tracemalloc.html

It is an order of magnitude slower and not less memory efficient. Also, for me it is never easier to understand what is going on when named tuples are used instead of dicts

Proof:

    def make_dicts():
        res = []
        for i in range(1000000):
            res.append({'x': i, 'y': i})
        return res

    def make_tuples():
        templ = namedtuple('Point', ('x', 'y'))
        res = []
        for i in range(1000000):
            res.append(templ(i, i))
        return res

    from ipython_memory_usage import ipython_memory_usage as imu
    imu.start_watching_memory()
    used 0.2305 MiB RAM in 4.00s, peaked 0.00 MiB above current, total RAM usage 86.41 MiB

    %time tuples = make_tuples()
    CPU times: user 780 ms, sys: 32 ms, total: 812 ms
    Wall time: 816 ms
    used 109.0625 MiB RAM in 0.92s, peaked 0.00 MiB above current, total RAM usage 195.47 MiB

    %time dicts = make_dicts()
    CPU times: user 160 ms, sys: 32 ms, total: 192 ms
    Wall time: 194 ms
    used 99.5117 MiB RAM in 0.30s, peaked 0.00 MiB above current, total RAM usage 524.33 MiB

Your sibling got opposite results. I haven't dissected either program, but I don't see any obvious flaws. I find your results particularly surprising. Can you explain them? Are there optimisations going on in your benchmark that you haven't accounted for?

It should not be controversial that namedtuples are clearer for the reason I stated in my initial comment.

sys.getsizeof() on a code object returns the size of the code struct, not all the other objects it references.

To measure real usage, you have to use another method. I use ipython_memory_usage because it's easy to use.

You can test this out easily by making a large dict and see how much your OS thinks python is using vs what sys.getsizeof is reporting.

- Dot syntax: foo.length rather than foo['length']

I gives a name and expected properties to common data structures, or unspoken unnamed types.

Many times when a dict is used, it's expected to have certain attributes, and may have expected ways of interacting with those attributes. Named tuples is a simple way to formalize these, which makes them not so mysterious to you future-you, or to other developers who inherit the project 6 months down the road.

When something is moderately awkward I often make a "snippet" in my editor, e.g.:

    pr<TAB> -->  print('%cursor%', %cursor%)
    nt<TAB> -->  namedtuple('%cursor%', ['%cursor%', …])

Be careful with named tuples:

    >>> from collections import namedtuple
    >>> Time = namedtuple('Time', ['hours', 'minutes'])
    >>> Point = namedtuple('Point', ['x', 'y'])
    >>> t = Time(hours=7, minutes=40)
    >>> p = Point(x=7, y=40)
    >>> p == t
    True

They're still tuples. They're meant more as a tool for making an already existing tuple mess slightly more managable rather than creating a new mess.

For making plain data objects take less boilerplate to create have a look at attrs: https://attrs.readthedocs.io

Strong +1 for attrs. It is a really nice library and quite easy to use.

Interesting find with attrs. My main use case for namedtuples is a class as a data container which doesn't require any special methods. This might be a better alternative.

Ugh, that's terrible... Thanks for pointing it out!

Classes work for this as well, I've found.

I use a lot of classes.

It feels like Java sometimes, but it does make debugging easier...

In my experience, namedtuples have significantly less overhead than classes.

Yes for the increased overhead, but I find that the fact that they're tuples underneath results in odd semantics (I think the comment above mine[1] does a decent job pointing out why I am leary of using them.

Tradeoffs, I guess.

[1] https://news.ycombinator.com/item?id=13124501

I doubt that because defining a namedtuple just creates a new class internally: https://github.com/python/cpython/blob/master/Lib/collection...

Class creation the very expensive in Python, so Guido van Rossum explicitly recommends against using dynamically created namedtuples: https://mail.python.org/pipermail//python-ideas/2016-April/0...

I believe that objects of namedtuple have less runtime overhead than a class. Guido seems to point out that the process of instantiating a namedtuple class is expensive.

Normally, yes. But if you define __slots__ on the class, it's the same overhead as a namedtuple (no __dict__ per instance).

I'd expect it to also improve things in regular code since class attributes are saved in __dict__, unless you use __slots__.

I might be wrong though (remember seeing that talk by Heroku about how objects are treated differently even in CPython).

Wow, the Python community sure is impressed with its small dicts. Have they tried pooling?

I'm disappointed in the interpolated strings. We finally get them, but its already behind the curve compared to Scala or C#, due to not being lazy and allowing context sensitive string interpolation.

The better design binds the values to the string, but does not interpolate into a string type fully. Then, the resulting object can be passed to a consumer like an SQL engine (and appropriately escaped) or an internationalization library (and swapped out for a translated string before interpolation).

I mean, I appreciate what they did, but it seems like missing a massive opportunity to provide a linguistic building block for library authors.

These ideas were discussed at length, but it was ultimately decided to keep things simple. Perhaps the mechanism can be extended in the future? In the meantime the 90% case is supported.

For i18n you can still do the same of _('your template here {foo}'), where _ is you gettext[1]. And once it has a lazy version, you will be covered.

[1] https://docs.python.org/2/library/gettext.html#gnu-gettext-a...

That doesn't work, unless `_` is a gettext overlay which goes and reaches into its caller stackframe and performs string interpolation itself. Otherwise you mean that you can "still do the same of"

    _('your template here {foo}').format(foo=foo)

which is a step down from

    f'your template here {foo}'

I do not know which feature I will miss the most whenever I have to use older versions of python, either the f-string interpolation or the underscores in numeric literals.

That's a joke, right?

can you please go troll somewhere else?

It seems to be just 3.6.0 RC1 actually: https://mail.python.org/pipermail/python-announce-list/2016-...

"The default console on Windows will now accept all Unicode characters and provide correctly read str objects to Python code. sys.stdin, sys.stdout and sys.stderr now default to utf-8 encoding."

Thank god.

My favorite bits include UTF-8 encoding support for Windows filesystems and console, scrypt, and OpenSSL 1.1.0 support.

Well, the "type annotation" system, where you get to type variables but the types don't do anything, was expanded again. It doesn't improve safety. It doesn't improve performance. The hand-waving argument is "Just as for function annotations, the Python interpreter does not attach any particular meaning to variable annotations and only stores them in the __annotations__ attribute of a class or module. In contrast to variable declarations in statically typed languages, the goal of annotation syntax is to provide an easy way to specify structured type metadata for third party tools and libraries via the abstract syntax tree and the __annotations__ attribute." Whatever that's supposed to mean.

I still suspect that was put in to screw over PyPy. If the type information was enforced, PyPy could use it to generate hard-typed code and improve performance. CPython, on the other hand, still represents everything as CObject objects in C. Type checking adds run-time overhead.

The Python interpreter itself may not do anything with the annotations, but static type checkers and linters can, and they will give you the safety that you need.

The annotations also allow the development of better IntelliSense-like tools.

this comment reads much better if you replace Python with Javascript and PyPy with V8 or whatever, especially with today's Typescript announcement.

additionally try using mypy-lang some. it's great.

Maybe this should be obvious, but I don't have a computer to hand to try and figure it out: In the example for f-string interpolation, why does value have to be declared like this:

  value = decimal.Decimal("12.34567")

Why wouldn't this work?

  value = 12.34567

Floats != decimals, and it's good practice to use Decimal I think.

It's not good practice. You must use Decimal when precision is important. Example: when dealing with currency, so arithmetic operations don't create or miss cents.

Because that gives you a Float, not a Decimal.

Doesn't seem very Pythonic to me...

Every type-inference language I am aware of assumes that anything that looks like a float is a float. It would be un-pythonic to assume that everything that looks like a float is a decimal. Imagine having to cast every damn float to a float.

In Haskell: `2.3 :: Fractional t => t`

In other words, it can be inferred to be any type that implements the Fractional typeclass, which includes Decimal types. Not that it is relevant here, because you basically need to have types for something like this to work, if you have proper type inference, then you don't need to assume any type, and can just interpret a literal as whatever is needed.

Since when does an exact decimal literal "look like a float" more than it looks like a decimal?

Tradition. It was easier for computers and the rest is history.

Yeah, I was going to say, "Since computers."

Sure, lots but not all (Scheme doesn't, for instance) programming languages treat things that look like decimals as floats as a popular performance-over-correctness optimization, but I don't think that makes an exact decimal representation look like anything but a decimal.

The Python Decimal class was introduced 10 years ago in version 2.4. That's after Python was 12 years old (from v. 1.0). It is a standard library module, not a first level type. This is not the least unusual.

Even Swift, as modern as it gets, operates the same way. You are arguing against a convention that is as deep rooted as CPUs capable of floating point calculations.

To this day, there ARE no "decimal" CPU operations. There are integer and float operations.

Actually, I am arguing against describing exact decimal representation as something that "looks like a float" (particularly a binary float.)

I'm not arguing against Python's behavior. I do think that, at a minimum, there are major and very common software domains where the common optimization is more harmful than beneficial, and that it's good that some languages buck the trend. But while I don't think it should be an unquestioned tradition in language design, I don't think it's categorically wrobg, either.

> To this day, there ARE no "decimal" CPU operations

First, this isn't true, there are CPUs that have decimal (still floating point, so only exact within a subset of their full range) operations.

Second, it's irrelevant; many languages have types (including types for common literals) that don't map to a specialized class of operations in the CPU, where the language compiles operations down to aggregates of operations on some lower level type or types.

Since about 1972 when C adopted that as a convention.

Sure, I get the difference between float and Decimal, but why can't floats be formatted with precision in the example? Are they using Decimal because it's best practice, even though it adds more code to the string formatting example ?

Floats can absolutely be formatted, but the result may be unexpected, because it is basically a (hardware based) C binary float:

https://en.wikipedia.org/wiki/IEEE_floating_point

So, because users will typically be surprised that 2.2 + 3.1 results in 5.300000000000001, or that round(2.675, 2) gives 2.67 (instead of 2.68), best practice is to use Decimal, which will give users the results they expect:

https://docs.python.org/3/library/decimal.html

Makes sense. I guess what I was getting at is that introducing Decimal to the example about f-string interpolation makes the example more complicated than it absolutely needs to be, but I can see why they did it.

I think some of that extra complexity could be papered over with either making decimal() available like int() and float(), which would eliminate the import, and/or implementing a decimal literal format, such as 0d2.675, analogous to the format of Hexadecimal and Octal literals (0xCC00CC and 0o267, respectively).

I like that second idea, perhaps a mention on the python-ideas mailing list?

Perhaps. I need to mull it over a bit before I take that plunge (but thank you for the encouragement!).

One possible objection is that while Decimal is in the standard library, it isn't a built-in type, so either Decimals need to be elevated to built-ins, or some magic needs to happen when importing the decimal module to add the literal.

They used Decimal just to show the nested attribute access

Here's a practical example that comes to mind. I work in film and we deal with frame rates. Often tools use 24 or 30 frames per second. 30fps is a simplification of 60 frames interlaced, which is technically 29.97fps. The main use-case we have is storing, comparing, and occasionally flipping between fractions and decimals. You would think integers would work. We've tried to update our tools to use 29.97 when it is necessary (although, this spans Python, databases, and third-party applications). The first attempt used floats, but we'd have scenarios where we'd get 24.00001 or errors comparing two values. The second attempt just used strings for frame rate so comparisons and storage work and you have a lookup table when you need to convert. Decimal sounds like a better solution.

A similar scenario in film (that hasn't caused as much trouble for me) is aspect ratio, which is the height/width of an image. Fractions and decimals are used interchangeably in conversation.

Besides using decimals instead of floats for your frame-rate use case (definitely a better fit), using a Python fraction may be a more useful representation for aspect ratios:

https://docs.python.org/3/library/fractions.html

Also possibly useful in that context (and bringing us back full circle to the OP's subject), Python 3.6 has added the as_integer_ratio() method to Decimal instances:

https://docs.python.org/3.6/library/decimal.html#decimal.Dec...

The nice thing about Python is that aliasing is pretty easy to shorten tedious statements (while not being magical). If I had to write tons of those in code, I might say:

  D = decimal.Decimal
  x = D("1.2345")

Also (at least this was once true) by doing so you save Python from a dictionary lookup after the dot "." each time.

Or from decimal import Decimal as D

You could, but it's very unobvious. Unless you had it as a clear convention across the code base I'd imagine it would just lead to a lot of confusion.

Wait. This is extremely common to do in Python. If it really led to lots of confusion, I don't think it'd be that common.

What's common isn't necessarily a good practice to follow - https://google.github.io/styleguide/pyguide.html#Imports

By importing a function and assigning a 1 char alias to it, the readability of the code is impacted.

I'm a bit confused by your comment. From your link: > Use from x import y as z ... if y is an inconveniently long name.

"import ... as" is both common and per Google's style guide a good practice to follow in this scenario.

No, the Style Guide says "use imports for packages and modules only", which (alias or not) rules out importing single functions.

You're right. I wouldn't import a function as a single letter module if I was only using it once or twice, but I do think there's value in some cases. decimal.Decimal("1.2345") or even d.Decimal("1.2345") can be cluttering.

Going by the guide, you would do 'import pprint; pprint.pprint("foo")'...which, at least in my experience, I've never seen. It's always "from pprint import pprint" or "from pprint import pprint as pp" It could be that I work with slobs.

I don't think having a different view on the relative value of conciseness vs. visible name spacing than is at play in the Google Python style guide makes you a slob.

You find "decimal" to be an inconveniently long name? In my experience "import...as" is reserved for things like PowerIterationClustering and the like. The overriding principle is to always choose readability over succinctness.

Depends how often it's used. If you're just calling it once I wouldn't bother, but I was going off of makecheck's comment of, "if I had to write tons of those in code." I misread the style guide linked, I do find decimal.Decimal("12.34567") inconveniently long--but I generally don't mind inconveniently long names for increased intelligibility and the IDE does most of the heavy lifting. I don't often use "as" it's usually to increase the readability.

Similarly, I often see "from pprint import pprint as pp"

While I haven't used it with something as rarely used as Decimal, I almost always alias numpy and tensorflow to np and tf respectively. I find it cleans up the code drastically.

Those are common conventions for those particular libraries though.

Maybe so, but more obvious than what the grandparent wrote.

Because the latter is a floating point number syntax.

There could be an argument for switching to decimals as standard, but even ignoring the practical consequences, I think for historical reasons (I assume this is the standard of programming languages; in Ruby for example is the same) it's not something that will be ever agreed/changed.

It's really neat to see async coming to things like generators and comprehensions. Looking forward to trying it out.

This will probably be an unpopular thing to say, and I understand that people have different opinions and preferences on this question.

However, for me, every time I see one of these Python 3 releases it makes me appreciate Go's model of language maintenance and development more and more.

Compare the last few major Go releases to the last few Py3 releases. Go has so much more discipline about when it's appropriate to add things to the language. Most of Go's changes are confined to bug fixes, performance improvements, and toolchain improvements.

By comparison, it seems like Python is on a runaway train of core-language additions.

Compare even PHP.

In that comparison, Go is a homeless (or maybe "minimalist") guy with a shopping cart.

There probably is a good balance between adding features and being conservative. I feel like Go is too minimalistic, making you have to write boilerplate code other languages like Python have nice abstractions for, while C++ is the opposite. But also, Python has been around a lot longer, so we'll see if Go keeps it's minimalistic philosophy over another decade.

this only proves the point that there's room for both. i like new features and can't wait for both f-strings and type annotations.

Althought I really like python and prefert it to go. I think what you say has a lot of sense. I would like to know why you are being downvoted. It's a valid opinion expressed in a nice way.

> PEP 515: Underscores in Numeric Literals

> PEP 515 adds the ability to use underscores in numeric literals for improved readability. For example:

    >>>
    >>> 1_000_000_000_000_000
    1000000000000000

A small touch but this is quite nice.

Love interpolation, still shaking my head @ the typing syntax

    primes: List[int] = []
    
    captain: str  # Note: no initial value!
    
    class Starship:
        stats: Dict[str, int] = {}

Could you expand? This looks similar to how it is done in Rust.

It looks very ugly to me, particularly the trying to reverse the original mistakes behind `int`, `list`, `dict` into `Int`, `List`, `Dict`.

If you want to fix that, you fix that first, then apply it to a subsequent patch like this one.

Additionally, I think Rust community, since its inception, embraced the goal of paying any up-front costs required for introspection.

Meanwhile, the Python community seems to be tearing between those who want syntactical beauty and one-way-to-do-it, and those who are running big production systems using IDEs like PyCharm and wanting to benefit them.

I'd prefer it if Python somehow warned people developing gigantic systems that maybe it's time to use a different language, instead of deforming Python to suit the people already running gigantic systems in a language unsuited to it.

Type hints are completely optional. Nobody is getting torn, and both use cases are equally valid.

see: "only one way to do it"

Well, I'm not going to stop using Python or anything. I just think they're bad and ugly and I'm sad that more and more I'll open the source code of some library I'm using and find those "hints" littered everywhere.

I would plea that people reconsider their inclusion, but it seems like that ship set sail.

This is probably one of the fairest criticisms I've read and, honestly, I didn't even notice that when reading the PEP. Thanks for your insight.

i had a similar opinion not long ago (as in, couple of months). then i was shown what mypy can do. it's really worth it.

I wonder if there's a way to make all strings be format strings?

Why would you want that?

I like strings to be interpolated by default, and that's one of the most annoying things about python.

Sure, but _why_? What benefits do you see that having over this? (Pre-pending an 'f' to a string)

Isn't "I like it" a good enough reason?

It never is, and if it was every language would be a hodgepodge of things some people liked at one time.

Yeah we call that PHP (and I say that as someone who uses PHP all day and quite likes it).

Oh, that's where it comes from. As a non-PHP user, I thought it was a strange feature.

No, it's much much older than PHP. It's in PHP because it was in Perl (which is also why it's in Ruby). I think iit was in Perl because it was in sh. But I wouldn't be surprised to hear that it's older than that. Snobol maybe?

I was actually thinking of PHP as I wrote that comment, I didn't want to call it out directly

I'm not asking for them to make it default for everyone. I just want it for me. So at a file level or something.

Maybe, but it is a disappointing reason. I imagine OP was hoping for some insightful, more objective reason for this.

Sure, but "I like it" is actually a good reason. You can dissect I like it into maybe that I find it more productive, that I find strings easier to read when they're like that, or for other reasons, but it actually boils down to aesthetics: I like it :)

It would break a lot of existing code.

Not by default. Just for me. Like "from format import default_strings" or something.

Really happy to see the secrets module being added to the standard library. Now to make sure every Python developer knows of its existence.

Hmm, the documentation seems to be updated to show 3.6 in the dropdown but it's not on the release page yet.

It's only a release candidate for right now.

Great, another release that python devs can refuse to use.

Ouch, harsh downvoting :) Guess the 2.7-4-life folks are a little sensitive. Anyways, it's great to see python continue to move forward. I just wish more devs would admit that python 2 is not the future and move on to 3.

> On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security

I prefer to use the phrase "less insecure" than "more secure".

I'll say it. Python 2: Stability. Python 3: Feature creep.

I've been doing a lot of Go lately and hugely appreciate the minimal syntax and orthogonal features. Seems like the opposite is happening not just in Python but many languages.

For me, it's Python 2: weird quirks (unicode support sucks, I have to do from future import for a lot of things, etc.). On the other hand, Python 3 fixes many of these things, and even others that I didn't realize could be better thought out (such as names of some stdlib functions, etc.)

I used to feel that way,but all my day jobs are python 2. They never to seem to want to budget in time to do an upgrade to python 3. It's been frustrating.

Lately I've been so frustrated I've actually been researching new languages with better performance, because that's a MUCH easy sales pitch to the business managers.

Julia :-)

PyPy3?

The time it would take to port code from 2 to 3 would likely be less than the time it would take to port to another language. And using the scipy stack makes numerical computing incredibly fast (fast enough for me considering it takes me longer to figure out how to analyze the results), though your use case may differ.

Are you the same person who advocates switching to Go over Python 3 in every single Python thread? Not everyone prefers the minimal approach, particularly in a dynamic, higher level scripting language, where ease of use trumps other considerations.

Go and Python occupy different niches, for the most part. So the design considerations for Go are naturally different than those for Python.

Nah that ain't me. I rarely comment on such matters actually. Like you said Go and Python serve quite different purposes and each do their jobs well. I use both on a regular basis and I've been using Python for over a decade. But I still think there's something to be said for simplicity in any language.

The vintage Scheme vs Common Lisp argument.

I'll say it too.

Python 2: Stability

Python 3: Features -- Unicode, yield from, better HTTP, asyncio, more flexible destructuring, better concurrency libs, better SSL/TLS

Python 2 is still a good language because they backported a lot of the improvements that landed in Python 3. It's headed in a good direction.

Reply on news.ycombinator.com