Hacker News Reader

Comment by paulddraper | Hacker News Reader

> As a developer who has primarily developed applications in Python for his entire professional career, I can't say I'm especially excited about any of the "headlining" features of 3.8.

Python is a fairly old, mature language.

What features would you have been especially excited about?

View on HN

The f-strings from 3.6 are a (relatively) recent feature that I have absolutely loved. I'd go so far as to say they are my favorite feature introduced by Python 3.

I'm also looking forward to PEP-554 [0], which allows for "subinterpreters" for running concurrent code without removing the GIL or incurring the overhead of subprocesses.

[0] https://www.python.org/dev/peps/pep-0554/

f-strings are great. Much nicer than ".format". I hope in a next iteration of the language all strings will be f-strings by default, avoiding the need to prefix them by a silly "f".

That would of course be an absolute disaster, given that any user input could easily leak internal state and/or break your program.

I meant for inline strings (i.e., typed by the programmer). This is an inoffensive change. The only possible ``accident'' is when the programmer wants to write "{x}" instead of the value of x. This is such and exceptional case that it may be best treated by forcing to escape the curly brackets.

If anything, user-input strings must be treated as tainted whatever the case.

That would break any existing strings that use curly braces, including most legacy use of str.format and docstrings that include code examples with dictionaries. It would be a big backward compatibility issue.

A quick look at the Python stdlib gives shows some of the breakage that would occur:

1) existing uses of .format() would break:

   aifc.py: raise Error('marker {0!r} does not exist'.format(id))

2) existing uses of "%" formatting would break:

    argparse.py: result = '{%s}' % ','.join(choice_strs)

3) many regular expressions would break:

    uuid.py: if re.fullmatch('(?:[0-9a-f][0-9a-f]-){5}[0-9a-f][0-9a-f]', value):

4) existing strings which contain uuids would break:

    uuid.py: >>> x = uuid.UUID('{00010203-0405-0607-0809-0a0b0c0d0e0f}')

Then you would run into the same problem Kotlin has with its raw strings – the brackets need to be escaped, but raw strings disable escaping, so you'd have to write r"{}" as r"{'\{\}'}" which is quite ugly and not very raw at all.

It's not an exceptional use case at all. f-strings are pretty much useless for any non-trivial formatting:

1) customizable templates (can't allow the user to access arbitrary variables)

2) i18n (can't allow the translator to access arbitrary variables)

3) complex values being passed to .format() instead of some variable

Explicit is better than implicit.

Sure, the curly brackets are explicit enough. What do you mean?

Sounds like PEP-544 is actually included in 3.8 on a "provisional" basis, which I guess just means they reserve the right to change the API. That's a killer feature for me, thanks for pointing it out!

Correction: So the PEP said it'll be in 3.8, but apparently it's been postponed to 3.9.

As someone familiar with the Python C extension API, I somewhat doubt that it'll make 3.9 either. Disclaimer: I'm only familiar with CPython as a user of the extension API; I have no idea on how the Python team plans to address the challenges I'm mentioning here, and what their current progress is.

The current extension module API encourages global state, e.g. types (and objects too) are allocated statically in a global C variable. For example, there is a global C variable `_Py_NoneStruct` that is the Python `None` value, and extension modules are accessing this variable directly.

Every use of this object needs to adjust its reference count, and that reference count is directly stored within the C global variable.

`_Py_NoneStruct` is currently even exposed in the PEP 384 stable ABI. Existing extension module binaries are commonly directly touching `_Py_NoneStruct.ob_refcnt` without any synchronization. Breaking the PEP 384 compatibility promise is fundamentally unavoidable here.

One of two things must happen: Alternative one: All refcount operations must be made atomic for thread-safety. These are really really common in the Python interpreter, but atomic operations are expensive on modern CPUs (especially if there's contention). But multiple threads using the value `None` would be quite common in Python code, so I doubt you'd gain any speed even at today's core counts -- in fact I'd expect the constant inter-CPU-communication for the refcounts to make everything slower than just using a single core with today's Python!

So alternative two, ensure no Python objects are shared between the subinterpreters. That's the plan. But that also means it's a breaking change for extension modules. And it's not just an ABI change (which would be handled by merely recompiling against the new headers). Any extension modules that do not yet support PEP 489 are already incompatible with subinterpreters, so that will take quite a bit of work until the ecosystem is upgraded. But there will probably also be some other breaking API changes. I think type objects are currently shared across subinterpreters, and those are frequently defined as a `Py_TypeObject` global variable in extension module code. Also, if every subinterpreter has its own GIL, extension modules calling `PyGILState_Ensure()` will have to specify which subinterpreter they will be using, so that the appropriate lock can be acquired.

My prediction: 3.9 may have the basic functionality, but it still won't be able to run on multiple cores concurrently. That will take a bunch of more work and breaking changes, and will likely be released as Python 4.0.

There will be another slow upgrade process ("my dependencies must upgrade before I can") until the Python ecosystem is multi-subinterpreter-compatible. But at least this one only affects extension modules.

I wouldn’t mind cpython performance improvements over language features. Specifically startup time. Py cli tools can easily take up to 300-600ms to start. Now, try to use them for scripting. A single script doing a bunch of calls to python clis has already several seconds of runtime penalty.

I completely agree, but you'll notice similar startup overheads in almost all non-native languages: Java, Ruby, Node.js.

I wonder if there is anything that can really be done, outside of something like Cython.

> I wonder if there is anything that can really be done, outside of something like Cython.

I compile my cli tools with nuitka [0], the resulting binaries take half the time to start. I find the difference quite notable.

[0]: https://github.com/Nuitka/Nuitka

The interpreter loads reasonably fast, takes around 20-40ms. And it's even faster if "import site on initialization" is disabled.

But startup time increases an order of magnitude once you start importing 'requests' and other common packages.

Perhaps an option to turn imports lazy...

Though not a language feature per say, I would love if the language would take on the packaging ecosystem and deliver a ground up approach that isn't just a kludge on a kludge on a kludge.

You might like portage. We have USE flags (which are ./configure --stuff), slots (so you can install multiple versions in a clean way), and any kind of dependency tree you can think of. When I am thinking of upgrading the system python, I can enable the next version and let things simmer for testing (building for py3.7 and 3.8 for example) before I actually throw the switch (eselect python set N).

Unfortunately, that'd mean a package manager for C, Fortran, etc. as well, since so many Python packages have non-Python dependencies.

I've started using Docker for ensuring prod behaves the same as dev. Going from Mac dev to Linux prod would otherwise cause trouble.

Sounds like Conda.

Conda can't fix some problems, like Python 3.7 wanting ncurses >6, but a dependency wanting ncurses 5.9

While not official, doesn't conda solve this problem? (Except for not containing the latest versions).

I've heard people recommend poetry on here as something that is good for package management, but I haven't tried it. Does anyone know of something like that, but also can build your application into a Docker container as well?

Into a docker container as well? The package manager isn't really related to docker though?

Does your container expose ports or need volumes? Does it need gunicorn or uwsgi in front of it? What about system packages? None of those (except maybe the last one) are really in the scope of the package manager.

The main use case is I have a project with a setup.py that has the normal stuff in it, and there's a main entry point in one of the files. I then make a dockerfile that installs that package in a container and runs the main file as the entry point. Ignoring things like ports, it would be nice to emit a standard dockerfile like that, since it's very common.

It's not great, but also it isn't terrible, the problem with Python packaging is that there were many ways to do it and people are confused. What's worse, there are tons of articles (in including from PyPA) that are providing bad information.

If you use setuptools and place all your configuration declaratively in setup.cfg it is not that bad.

I'd really like a way to refer to a variable in a outer scope (like global... but not just for global variables).

That would be “nonlocal”. https://www.programiz.com/python-programming/global-local-no...

In the same vein, they could add syntax to distinguish introducing a variable from setting the value of an existing variable.

Python's heuristics for that are pretty annoying.

I think that feature has exactly zero chances of getting adopted, but that's probably one of my top Python annoyances too.

If it's any consolation, the scope of a variable isn't determined by heuristics, it's just that the rules are (IMO) kinda bad. Basically, if a variable is assigned (in addition to `=` and friends, `import`, `class` and `def` are also assignments in disguise) to in what Python calls a "block" (which is not like a C-style block; rather `class` body, `def` body, or top-level) it is considered to belong to that block (with the notable exception of the name for a caught exception in a `except Exception as e: ...`, see [1]). If you want it to refer to a variable outside the block, you need to use `global` or `nonlocal` as appropriate. The full gory details are in [2]

1: https://docs.python.org/3/reference/compound_stmts.html#the-...

2: https://docs.python.org/3/reference/executionmodel.html#nami...

Yes, I meant the rules are bad. I called them heuristics, because I assume they came up with the rules as a heuristic to what would be the most useful behaviour.

Python didn't use to have lexical scoping, so the rules are retrofitting around that.

Yeah, it all seems a bit hacky. Especially the bit where a caught exception bound to a variable is cleared at the end of the `except` clause. I'm pretty sure it's because if you didn't you'd get a reference cycle (exception object -> stack trace -> function invocation record -> function locals -> exception object), which makes CPython's GC sad.

... and what prevents you from doing that? You mean you'd want to reassign them?

Yeah the fact that a new release is boring is a good thing.

Multi-line/multi-expression lambdas.

"end" statement which would enable automatic indentation.

"switch" statement instead of "if/elif/.../else" hell.

Oh yes, too all of them!

Correctly done multi-line lambdas/expressions could bring Python's expressivitiy to where ECMAScript is nowadays, where you can write both foo = function(...) or function foo().

Perl 6 has a great switch statement, called "given" https://docs.perl6.org/language/control#given

A null coalescing operator

There was a now-deferred PEP for 3.8 https://www.python.org/dev/peps/pep-0505/

I would honestly like an option to "compile" python to check for typing inconsistencies with the type hinting introduced in Python 3. Also, the multithreading story is kinda shitty

> I would honestly like an option to "compile" python to check for typing inconsistencies

You can use mypy for that: https://github.com/python/mypy

Pattern matching. Quasiliterals. PSF support for Poetry or PyPy.

I would love if they added "?" to the list of valid characters for names, so I could write something like `valid?()` instead of ``is_valid()`.

just use "_P", a fairly-widely-recognized substitute from the Lisp world.

As a python programmer I would have no idea what `valid_P` is supposed to mean. Valid Pascals?

P is shaped like a question mark, though I think it was retronymed to stand for "Predicate"

I think "predicate" is the original meaning of that postfix.

Emacs Lisp, at least, can use question marks in names without batting an eyelash.

I see no reason that would not have been true for Lisps generally since their birth.

The suffix is just "p" in Common Lisp[1], so I think the "predicate" explanation is more likely to be correct than the visual analogy to the question mark.

[1] http://www.lispworks.com/documentation/HyperSpec/Body/f_list...

Ruby to the rescue!

Or even better, just `valid?` Oh, wait .... that's Ruby.

I want to build executables that I can pass around without also sending an entire environment. Crazy that’s not a native feature.

Honestly? Performance and the possibility of making Pypy the reference implementation.

proper threads.

Reply on news.ycombinator.com