What’s New in Python 3.7 | Hacker News Reader

What’s New in Python 3.7

docs.python.org

174 pointshappy-go-lucky67 comments

Unrelated to python 3.7, but they recently added this feature that you can add two dictionaries together in 3.5 by doing:

  new_dict = {**dict1, **dict2}

It is so handy and nice

I didn't know about that. It's also unbounded:

    >>> d = {**{1:2,2:2,3:2}
            ,**{4:2,5:2,6:2}
            ,**{7:2,8:2,9:2}}
    >>> d
    { 1: 2, 2: 2, 3: 2
    , 4: 2, 5: 2, 6: 2
    , 7: 2, 8: 2, 9: 2}

What happens to duplicate keys?

They get overwritten by the value of the dictionary that came last.

  >>> dict1 = {"key1": 1, "key2": "hello", "key3": [1, 2, 3]}
  >>> dict2 = {key1": 1, "key2": "world", "key3": [1, 4], "key4": "4"}

  >>> {**dict1, **dict2}
  {'key1': 1, 'key2': 'world', 'key3': [1, 4], 'key4': '4'}

Lots of code to show that. How 'bout this: ;-)

  >>> {**{'a': 'first'}, **{'a': 'second'}}
  {'a': 'second'}

I think it raises an exception.

https://www.python.org/dev/peps/pep-0448/

EDIT: yeah, nvm, it's what everyone else said.

No it doesn't, you simply get the value from the last dict:

  >>> a = {'x': 1}
  >>> b = {'x': 2}
  >>> {**a, **b}
  {'x': 2}
  >>> {**b, **a}
  {'x': 1}

Last pair wins, same as setting an existing key or doing an update() with an existing key.

Looks like this came from https://www.python.org/dev/peps/pep-0448/

Seems similar to the spread operator in JavaScript: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Why doesn't the + symbol work?

+ is not a supported operand for dict. You can probably subclass dict and add the plus operand

Why add that crazy syntax when a + would be a lot clearer?

It makes sense in context. {} is a dict literal, and * * dict is already used other places to expand a dict into key-value pairs.

Given

  a = {1: 1}
  b = {2: 2}

This

  c = {**a, **b}

is equivalent to

  c = {1: 1, 2: 2}

a|b would be the obvious choice, because it already works for set:

    {1,2}|{2,3} == {1,2,3}

I guess the reason they went for the weird {∗∗a, ∗∗b} syntax is that it is not a commutative operation.

I guess I didn't explain it clearly enough. ∗∗a is not new syntax, it's just applicable in a new context.

Wouldn't mind that either.

Correct. This is called orthogonality. I just used it now to discover that this same syntax works for both sets and lists (using * instead of )

  >>> s1 = {'a', 'b', 'c'}
  >>> s2 = {1, 2, 3}
  >>> {*s1, *s2}
  {'b', 1, 2, 3, 'c', 'a'}

  >>> l1 = ['a', 'b', 'c']
  >>> l2 = [1, 2, 3]
  >>> [*l1, *l2]
  ['a', 'b', 'c', 1, 2, 3]

It's fun that I use it every day but my linter still thinks is a syntax error :)

A linter for a dynamic language has got to be the hardest thing ever to implement.

In PyCharm if you return from within a contextlib.suppress context it'll think any code following it is unreachable code.

https://youtrack.jetbrains.com/issue/PY-25809 https://youtrack.jetbrains.com/issue/PY-16419

Ah, great. This used to be rather clunky to do.

Really? This is how that was done in Python 2.7 and it doesn't seem that much clunkier to me...

    dict(dict1, **dict2)

That's only allowed if the keys in the second dictionary are valid Python identifiers (don't contain spaces etc.). I think CPython is OK with this, but PyPy is not.

It has a lot of issues. The biggest one for me is the following: Dicts are intended to take hashable keys (e.g. frozensets or tuples), but this method fails in Python 3 when keys are not strings.

Source: https://stackoverflow.com/questions/38987/how-to-merge-two-d...

Interesting. I never thought of that. I also asked a question about this on StackOver flow once and no one suggested that. Very useful.

Some nice optimisations included:

    Added two new opcodes: LOAD_METHOD and CALL_METHOD to avoid instantiation of bound method objects for method calls, which results in method calls being faster up to 20%. (Contributed by Yury Selivanov and INADA Naoki in bpo-26110.)
    Searching some unlucky Unicode characters (like Ukrainian capital “Є”) in a string was to 25 times slower than searching other characters. Now it is slower only by 3 times in worst case. (Contributed by Serhiy Storchaka in bpo-24821.)
    Fast implementation from standard C library is now used for functions erf() and erfc() in the math module. (Contributed by Serhiy Storchaka in bpo-26121.)
    The os.fwalk() function has been sped up by 2 times. This was done using the os.scandir() function. (Contributed by Serhiy Storchaka in bpo-25996.)
    Optimized case-insensitive matching and searching of regular expressions. Searching some patterns can now be up to 20 times faster. (Contributed by Serhiy Storchaka in bpo-30285.)
    selectors.EpollSelector.modify(), selectors.PollSelector.modify() and selectors.DevpollSelector.modify() may be around 10% faster under heavy loads. (Contributed by Giampaolo Rodola’ in bpo-30014)

edit: That one around the unlucky Unicode characters is fascinating. https://bugs.python.org/issue24821

Due to the way that regex searching is optimised, it trips up because: ".. the lowest byte of the code of Ukrainian capital letter Є (U+0404) matches the highest byte of codes of most Cyrillic letters (U+04xx). There are similar issues with some other scripts ..."

"More than 255 arguments can now be passed to a function, and a function can now have more than 255 parameters."

There are human devs who needed this? Or is this a sign that AI bots are now involved in language design?

They have issued tagged with the bug report and adding more arguments to named tuple and to general functions seems to be the issue. The motivation for some of these changes were due to automatic code generation.

See https://bugs.python.org/issue18896 and https://bugs.python.org/issue12844 for more information.

Also, Django 1.3 had issues with 255 arguments for named tuples.

To my knowledge there are no "AI Bots" that are involved in language design. Code generation is possible. Also solver aided languages See https://emina.github.io/rosette/

Saw one of those in a vb.net shop. It went that way:

- someone creates a function to do something on a business concept that was not materialized by a class. Think of a bunch (~ 8) of properties that belong together but for some bad reason are moved in separate variables all over the place

- another function has to work on a set of those objects. The dev who implements it thinks it would be a good idea to keep all parameters separate. No we have nproperties * nobjects parameters. The dev use a rigorous naming convention for readability (creatorFirstName, creatorLastName, validatorFirstName, etc)

- now we have multiple sets (creatorFirstName1, 2, 3 ...)

I don't remember the details but there was around three hundred parameters.

edit: formatting

On python-dev a few month ago, Guido agreed with you that this is an arbitrary limitation that should be removed at some point. In particular, he worried that programmatically generated code would tend to run into this limitation.[0]

[0]: https://bugs.python.org/msg142998

First thing that pops to mind is ORM for extremely column-wide tables, which are not unusual in some domains.

I'm not too familiar with Python but something like that would typically be handled by passing a pointer to a data structure containing the column data, and not each column itself (or does/did it have a 255-item limit on its other data structures too...?)

OK, but then you need to populate the data structure. You can do it with individual member assignments of course, but some quick Googling (I am not a Python expert either) suggests that the default row/object constructor for SQLAlchemy is an arg-per-field function.

Perhaps it could trigger if you do function(*iterable), with a huge iterable?

Nope, that works just fine.

  >>> def f(*x): print(x[-1])
  >>> f(range(300))
  299

And surprisingly, in python 2.7, I was able to define a function that takes more than 255 arguments. But perhaps this only worked because I cheated and used exec.

  >>> exec("def f(" + ",".join("f" + str(x) for x in range(300)) + "): print(f299)"
  >>> f(*range(300))
  299

Interestingly enough, the function def (the exec part) does not work in 3.6, as one could expect, but in 2.7 even though the function definition works, you can make the call fail by generating actual parameters:

  >>> exec("def f(" + ",".join("f" + str(x) for x in range(300)) + "): print(f299)")
  >>> f(*range(300))
  299
  >>> exec("f(" + ", ".join(str(i) for i in range(300)) + ")")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<string>", line 1
  SyntaxError: more than 255 arguments

When reducing the number of parameters to be invalid for f, the following error is shown:

  >>> exec("f(" + ", ".join(str(i) for i in range(30)) + ")")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<string>", line 1, in <module>
  TypeError: f() takes exactly 300 arguments (30 given)

Which seems a bit paradoxical - "You need 300 arguments" - "No, wait, actually, you can't have more than 255" ... ;)

It looks like they either had changed the internals of python 3, making it fail at the definition as a side effect instead of when calling, or used the same underlying logic but purposely chose to issue the exception to potentially catch the problem earlier...

I think *iterable would just pass one argument that's a list, not pass each argument individually

There may be some optimization at the interpreter level, but it appears to Python that each element of 'iterable' is passed as an individual argument

    >>> print([1,2,3])
    [1,2,3]
    >>> print(*[1,2,3])
    1 2 3

That doesn't tell you anything about when the list is expanded.

The asterisk at the call site (where it turns an iterable into multiple passed arguments) not the function definition site (where it collects otherwise unconsumed arguments into a single list.)

I believe args and *kwargs have special opcodes already.

Could be useful for code autogeneration of some ilk.

Guido had stated that he didn't want the implementation to place arbitrary limits on the language specification.

This particular limit was not present in the early Python 2 releases.

When the limit was added, limiting arguments wasn't the goal; instead, it was just an artifact of a patch regarding how keyword arguments and positional arguments we encoded in Python's bytecode scheme.

Dear God why.

Switching the default encoding from ASCII to utf-8 sounds like a pretty big change.

Isn't ASCII a strict subset of UTF-8?

If anything this should make sure programs don't break because they read some utf-8 extended character when expecting just ascii -- which is the most common case.

I didn't say it's a bad change!

But since you raise the issue, there are downsides: Many programs developed on 3.7 will inadvertently break on pre-3.7 Python versions. As we know, this is a big thing on parts of the Python ecosystem because users frequently rely on the Python version that come with their OS or framework (eg AWS Lambda), and many developers want to stay compatible with the most common platform-shipped Python versions.

It's a good assumption for 3.7. In isolation this is a pragmatic decision and makes sense.

You are correct though. A .py file which uses non-ascii and doesn't declare the encoding should be considered an error though. Hopefully everybody's dev tools will pick this up (even on 3.7 where they wouldn't need to)

Not really, imho. It makes more things work out of the box, but input and output for things that used to be ASCII anyway remain the same.

The only real question is:

When am I going to be able to name my variable as emojis?

Well Python 3 broadly follows UAX 31 "UNICODE IDENTIFIER AND PATTERN SYNTAX", and emoji don't have the xid_start or xid_continue properties, so… never unless you decide to be the change to believe in and get the Unicode consortium to change its ways?

You've been able to do that for a while.

edit: I take that back. Apparently thebunicode allowed in identifers is not unrestricted..... in my defense I don't use unicode identifiers as a rule....

You can use non-ascii characters but not emoji as arbitrary symbols are not generally classified as either XID_Start or XID_Continue.

When Apple releases a MacBook Pro emoji keys only

> Support for building --without-threads is removed. (Contributed by Antoine Pitrou in bpo-31370.).

Woooo! That means, at some future time, I'll be able to remove code checks for non-threading Pythons.

This patch was a little aggressive and may get reverted (it caused immediate code breakage in downstream distros and projects). Changes like this typically need a deprecation period (that said, to the extent the current feature didn't really work in the first place, we can immediately take it out).

What kind of checks did you use?

What version will contain dataclasses?

What do you mean by this? Do namedtuples not give you this?

  >>> class MyClass(namedtuple('_MyClass', ['x', 'y', 'z'])):
  ...   def foo(self):
  ...     return self.x * self.y + self.z
  ... 
  >>> x = MyClass(4, 5, 6)
  >>> x.foo()
  26

... or an attrs decorated class with frozen=True?

  >>> import attr
  >>> @attr.s(frozen=True)
  ... class MyClass:
  ...   x = attr.ib()
  ...   y = attr.ib()
  ...   z = attr.ib()
  ...   def foo(self):
  ...     return self.x * self.y + self.z
  ... 
  >>> x = MyClass(4, 5, 6)
  >>> x.foo()
  26

Close! ericvsmith, gvr, et al were inspired by attrs and have since been implementing a stdlib version of it, calling it "dataclass"

see: https://github.com/ericvsmith/dataclasses

and this comment thread is revealing: https://github.com/ericvsmith/dataclasses/issues/19

Thanks for that link. Very informative interesting discussion. There must be a lot of desire for attrs from the stdlib developers.

Reply on news.ycombinator.com