Things to use in Python 3 | Hacker News Reader

Things to use in Python 3

datawhatnow.com

581 pointsausjke255 comments

> Type hinting (3.5+)

> Static vs dynamic typing is a spicy topic in software engineering and almost everyone has an opinion on it. I will let the reader decide when they should write types, but I think you should at least know that Python 3 supports type hints.

I think this gives readers the impression that type hints have anything to do with dynamic vs. static type systems, which isn't true. These are merely annotations that are attached to values. In fact, it's legal to use full strings as type annotations — they have no inherent semantic value.

You can gain semantic value by using a tool like MyPy or an IDE like PyCharm, but the type annotations do not in themselves do anything. I think it could be worth clarifying this for readers who are unaware.

How to do Type hinting with argument default values?

Pretty straightforwardly:

    x: int = 0

This makes sense when you see that the variable production in the Python grammar [0] has been updated to essentially read "either a lone variable or a variable with a type annotation". (I highlighted the relevant line of the grammar for you in the link.)

[0] https://github.com/python/cpython/blob/master/Grammar/Gramma...

Except further optimisation in Cython like http://docs.cython.org/en/latest/src/tutorial/pure.html#stat...

I think this falls under the realm of "gain[ing] semantic value by using a tool like MyPy". The implementation of static typing by way of processing type annotations is an implementation choice, and not a part of the Python language specification.

The implementation of static typing by way of processing type annotations is an implementation choice, and not a part of the Python language specification.

That comes close to being the worst idea in the history of programming languages. Types aren't checked, you can't trust the annotations, and you give up a big optimization opportunity. But Guido's naive interpreter will still work.

If you say something is a float, the compiler needs to both enforce that and use that. Then maybe the Python crowd wouldn't need to call out to C code whenever they needed to do some number crunching.

Why is calling out to C code via something like numpy a bad thing?

Python is never going to be as fast as C. Ever. Even if you somehow managed to decouple numeric types from PyObject in a backwards compatible way it would not be as fast.

Type Hints can’t be used as an optimization in the interpreter due to the way they are resolved. And in any case, it would not be safe to do so and there are very few cases where optimizations make sense.

>That comes close to being the worst idea in the history of programming languages. Types aren't checked, you can't trust the annotations

As for types not being checked, that's not a problem. JITs do extra specialization on assumed types even without annotations all the time, and just drop them at runtime when they are not valid anymore (e.g. a variable that only pointed to integers now stores a string). Still, this runtime overhead of dropping specialized code aside, those optimizations make e.g. v8 hella fast.

(And of course you could also just enable optimizations after you've statically checked the whole program types with something like mypy -- this is even easier and less dynamic than what v8 does).

>and you give up a big optimization opportunity

You don't give anything up, since an implementation can take advantage of this "big optimization opportunity" if it wants and has the manpower to add it.

>But Guido's naive interpreter will still work.

It's also less work, which unless we have a volunteer here, was and remains the intention.

> Then maybe the Python crowd wouldn't need to call out to C code whenever they needed to do some number crunching.

Numpy and Scipy use a lot of Fortran at the backend, and unfortunately, beating Fortran when it comes to number crunching is insanely difficult, not just because of the language, but because those libraries like LAPACK have spent decades being refined into incredibly fast and accurate systems.

Python's ability to talk to those libraries is a bonus. But Python itself could never compete directly with them. Pure-python code isn't capable of some of the insane speed those libraries can pull off.

>because those libraries like LAPACK have spent decades being refined into incredibly fast and accurate systems.

In addition, due to their use in benchmarks, x86 has in part been designed to make LAPACK fast.

Though nothing prevents it from becoming "part of the Python language specification" -- except the backend implementation (besides Python doesn't have a specification, it's what the CPython does).

Well I think you could argue that Python is specified in part by CPython, but also in part by the PEPs. As static type checking is inherently opposed to the Python philosophy, it would take a PEP to force CPython to change its implementation accordingly.

Thanks for pointing that out! -- I've been missing a feature like that in Numba (which is, otherwise, really cool).

Yeah, I'm pretty sure Python itself explicitly uses the term "annotations" everywhere instead of "type hints". The point is that they can be used for anything, typing being one such use.

As for libraries, other than MyPy, there's also Pyre by Facebook and Pytype by Google. I'm definitely excited to see more advanced static analysis being implemented beyond simple type checking. Some of these libraries are starting to explore that.

Permitting arbitrary annotations is a rather nice feature that, besides static analysis, opens the door for expressive runtime pre/postcondition checking functionality like the Clojure spec library[1]. Does Python have something similar?

[1] https://clojure.org/guides/spec

https://andreacensi.github.io/contracts/tour.html#quick-tour looks to be a stable solution for that.

Unfortunately PEP 563 [1] means that any non-type-hint usage of annotations is considered deprecated. You could still do things with decorators (like mentioned in a sibling comment), although I suppose that doesn't offer such an elegant syntax.

[1]: https://www.python.org/dev/peps/pep-0563/#non-typing-usage-o...

Although not really what they were intended for, somebody used annotations to provide a macro system [0].

[0] https://tinkering.xyz/abusing-type-annotations/

Author here. Be warned, this was just a proof of concept that I did because I thought it would be fun/weird/interesting!

Oh my goodness. I've committed some naughty deeds with the ast module before, but this is truly, wonderfully, heinous. You push this idea so far that I wouldn't even recognize some of these examples as syntactically valid, let alone predict their effects. I'm awestruck and a little frightened of you.

The docs at python.org refer to them as “type hints” ️

https://docs.python.org/3/library/typing.html

`typing` is the module that uses annotations as type hints. Parent is correct; Python documentation calls the syntax annotations.

So Python docs refer to both depending on the context.

As mentioned in the other comment, this is just a library used to give type hints as annotations. The feature itself is called annotations, this is one specific use-case.

The reason this library exists in stdlib is so that all the different type hint libraries can interoperate. Otherwise if they each brought their own typing, you couldn't switch between different static analysis libraries.

They are available in the runtime, so you can do runtime reflection on the annotations and perform dependency injection if that is your thing. In that sense they do something: they provide information to the runtime which you can use if you want.

I think this is just a semantic argument at this point, but to me what you mention isn't evidence of the annotations doing anything. They make it possible for things to be done, just as I said.

The distinction I'm drawing is to statically-typed languages where simply adding annotations will actually change semantics of the program (e.g., by forcing a downcast or something).

But you're absolutely right that annotations can be useful for many things!

The most underrated part of type annotations are so-called stub files [0], which allow you to keep your annotations separate from the actual code, thus reducing its size while still benefiting from the static analysis with mypy.

Unfortunately the IDE support for those files (.pyi extension) is quite limited -- I have even suggested to the PyCharm team to implement an option to automatically save all annotations to stub files, while still displaying them alongside the code in the editor.

[0] https://www.python.org/dev/peps/pep-0484/#stub-files

I'm sort of sad that Python has reinvented .h files.

Same here. Fortunately those files are entirely optional and apparently seldomly used, judging by the limited IDE support after 5 years.

Whoa, that's neat! I had no idea!

Does PyCharm support these? Like if I write the stub files manually, will type hints and autocompletion work correctly in the main file? This could be a great way to reduce the amount of syntactic overhead, at least in function definitions and top-level code.

At the moment, you can click on a (say) class definition, and, if it has the corresponding entry in the pyi file, it will open it. But I would like it to be a bit more integrated.

Another advantage is that you can use pyi files for static checking, but remove them when deploying the application.

Sounds sort of like the .mli files in Ocaml. As I understand it, the compiler uses (and enforces) the types found therein, though can infer a lot from the code in .ml files as well.

Are there any tools that can use the annotations and watch a running script and warn you if they’re violated? Maybe during your test suite.

Or linters could do some basic analysis.

off the top of my head there's pytypes¹ and typeguard² – both look at type annotations and enforce them at runtime.

¹ https://github.com/Stewori/pytypes/

² https://github.com/agronholm/typeguard

What's the best practice for type hinting a class that's not being imported?

I.e. A module contains a function that accepts a pandas.DataFrame. This module doesn't import pandas.

from typing import TYPE_CHECKING

if TYPE_CHECKING:

  import whatever

def foo(param: 'whatever'): ...

Thanks! :)

F strings have been amazing in my everyday programming. I have been programming in Python for about 8 years now. I'd say some additions to this list are

  Conda for managing environments and Deep Learning builds such as Cuda on Ubuntu
 
  Jupyter notebooks for quick prototyping

  IPython if you have never seen it before.

  Requests Library

Before I knew about F strings my .formats() were just so long. F strings are nice for sure

Despite being so cool, it's also a very under utilized feature, as it inherits all the formatting capabilities of format().

With format(), and hence with f-string you can:

# Replace datetime.stftime()

>>> f"{datetime.now():%m/%d/%y}"

'05/15/19'

# Show numbers in another base

>>> f"{878:b}"

'1101101110'

>>> f"{878:x}"

'36e'

>>> f"{878:e}"

'8.780000e+02'

>>> f"{878:o}"

'1556'

# Control the display of padding, filling, and precision of numbers

>>> 1 / 3

0.3333333333333333

>>> f"{1 / 3:.2f}"

'0.33'

>>> f"{69:4d}"

' 69'

>>> f"{69:=+8d}" '+ 69'

>>> f"{69:0=8d}"

'00000069'

# Center text

>>> f"{'foo':^10}"

' foo '

And so many other things, since you can create your custom __format__.

Not affiliated, but in this regard, I quite love https://pyformat.info/ as a format cheat sheet.

> I quite love https://pyformat.info/ as a format cheat sheet.

I never knew this existed but that's awesome! Thanks for the link!

To this novice, the "Old" way looks so much better.

What old way ? %, zfill, strftime, or some hack with len() ? Having a unified syntax with all formatting needs is a big deal.

This is maybe not directly visible in the parent comment, and particularly not on the website, that is compatible with old versions of python also, but to me there's a huge difference in usability also in the ability to use variable names in the current scope without additional verbose crap by default. This seems secondary to a lot of people, but to me that changes everything:

  >>> x = 1
  >>> f"{x}"
  '1'

This is much more readable than:

  >>> "{x}".format(x=x)
  '1'

  >>> "{}".format(x)
  '1'

It makes the code immensely more readable than having to count parameters, especially for long strings with a lot of data in them. Counting is for computers, not programmers; I shouldn't have to count anything for such a trivial task.

So, thanks to f"", no need for what I find to be an ugly additional call to .format(...), that is just visual clutter for most cases.

The only reasons I could see for a call with a dedicated dict is data exfiltration from a user provided format string executed on python code they do not control, or renaming/subsetting of keywords for cleanliness. Both are special cases, and that's the effect the current implementation has, thankfully.

I disagree, especially when using named parameters. The new style is too implicit.

How is it implicit? Every insertion is expressed in exactly the place where it will print. It's way more explicit than having a placeholder and a value on completely different parts of the expression.

> It's way more explicit than having a placeholder and a value on completely different parts of the expression.

No its not.

  "My name is {}, I'm {}'{}'' tall and live in {}".format(name, location, feet, inches)

This error is much better hidden than in the equivalent:

  f"My name is {name}, I'm {location}'{feet}'' tall and live in {inches}."

I did mention named parameters, you are using positional parameters to make your point. You don't even list the variables in your example. That seems pretty implicit to me.

I think this is more explicit:

"My name is {name}, I'm {feet} tall and live in {location}.".format(name="Bob", location="USA", feet=7)

Your example is not representative because you're using literals.

Your actual example would be:

    "My name is {name}, I'm {feet} tall and live in {location}.".format(name=name, location=location, feet=feet)

And with f-strings:

    f"My name is {name}, I'm {feet} tall and live in {location}."

f-strings is clearly more readable, concise and actually faster to run.

The literals were just an example. And personally I find it more explicit, which is my original point.

If its faster to run, then fair enough.

Using your example, I think

"My name is {name}, I'm {feet} tall and live in {location}.".format(name="Bob", location="USA", feet=7)

is less explicit than

"My name is Bob, I'm 7 feet tall and live in USA."

Yes it is.

and that last one is faster to run ;-)

I see your point, but the syntax inside of fstrings forms an entirely new sub-language (DSL). I can look at old python and know exactly how the .format() calls parse with positional and named keyword arguments, but looking at the fstrings I need to be familiar with all of the new magic.

I've heard similar complaints about the loop macro in Common Lisp, and this feels a bit like shell or Tcl strings which could mean anything (arguments to sed/awk or paths to widgets/urls). However, I guess people are familiar with regular expressions as a completely foreign nested language (DSL), so this isn't much different than that.

I agree, it seems to be getting away from "explicit over implicit".

I love F-strings, my only gripe is the chosen syntax for invoking them. For starters, F may or may not be capitalized, so these two lines are identical.

  F"My string with {interpolated_value}"
  f'My string with {interpolated_value}'

Like using apostrophes or quotation marks to enclose strings (or F-strings), I find myself wasting time questioning the trivial notion of whether to capitalize or not.

I think Groovy's syntax is great, where quotation marks denote an interpolated string, and apostrophes denote a plain old string.

  "This is my ${interpolated_value} string"
  'This is my plain old string'

But these are nitpicks, and it's really too late to implement something like that.

> I think Groovy's syntax is great, where quotation marks denote an interpolated string, and apostrophes denote a plain old string.

That's Perl's syntax, from about 15 years before Groovy, whippersnapper. Off my lawn, now.

Yeah, Groovy got it from Ruby which got it from Perl.

Yeah, Ruby uses the doublequote/singlequote distinction from Perl, but Ruby tags variables to interpolate with #{}, while Groovy and Perl use $.

Oh, although I guess Perl got that syntax from the Bourne shell…

> Yeah, Ruby uses the doublequote/singlequote distinction from Perl, but Ruby tags variables to interpolate with #{}, while Groovy and Perl use $.

Huh, I could have sworn that I remembered that Groovy has taken it from Ruby.

Incidentally, Ruby uses just # for variables, it uses #{} for arbitrary expressions. A common style prefers #{} even where # alone works, though.

#notallvariables

Apache Groovy's string syntax is deficient, not great. It doesn't have the docstrings syntax that Python, Ruby, and Perl all use.

An early beta version of Groovy 1.0 had it (thanks to a Sam Pullara), but it was later yanked out. Groovy's self-styled Project Manager at the time said he only wanted syntax in Groovy that would cause the Java syntax highlight rules in Eclipse and Netbeans to highlight Groovy code similar to Java, so if a manager was walking around the programming area, the screens would look like the programmers were using Java.

And that's how Groovy got its deficient string syntax.

I had a co-worker who like the convention:

  >>> 'User {user} has logged in and did an action {action}.'.format(**locals())
  'User Jane Doe has logged in and did an action buy.'

I thought it was clever at first, but noticed it was frustrating to refactor because it was hard to see the scope of variables (especially if the template string is defined elsewhere). I don't lean heavily on IDEs, but at the time it would flag many required variables as unused because it couldn't grok this.

Do you find the same problems when using f strings?

    >>> 'User {user} has logged in and did an action {action}.'.format(**locals())

To me, this is pretty gross. This implicitly requires all the correct variables to be bound in the local scope, and there's no easy way to check that this is the case without reading all of the code leading up to this point (which is bad, in my opinion). Because what this really does is create string-level variables `user` and `action` and pass the values of similarly-named (i.e., shadowed) variables in the outer (but still local) scope in their place. This is why IDEs struggle with this: they can't simply know the contents of the dictionary returned by the function `locals()` without actually executing the code (unless they special-case calls to functions named `locals`, but this causes problems if the user ever shadows this name).

F-strings directly use the bound variables in the outer (but still local) scope:

    >>> f'User {user} has logged in and did an action {action}.'
    'User Jane Done has logged in and did an action buy.'

This avoids the shadowing problem, thus making it easy to use with IDEs and (IMO) easier to grok while reading directly, although maybe most people don't think about shadowing to the same extent I do and wouldn't be as bothered by this in general. Still, the IDE support is incredibly useful.

there are python implementations where locals() is unavailable or available at significant cost.

> Do you find the same problems when using f strings?

No, you can't define f strings elsewhere, and in theory your IDE can statically analyze them just as the Python interpreter does. In practice, your IDE might suck, I don't know.

They just added a brand new feature in 3.8 that let's you just append = at the end for quick debugging:

>>> f'{5+5=}'

'5+5=10'

>>> foo = 5

>>> f'{foo=}'

'foo=5'

https://docs.python.org/dev/whatsnew/3.8.html#f-strings-now-...

This just feels like another example of Python slowly creeping more esoteric syntax into the language, making it harder for beginners to approach the language.

Beginners don't have to know or touch any of this though. They can keep using normal strings until at some point they learn about f strings and see a need for them. You don't need to stop adding features to a language just because a beginner won't be able to learn everything. They don't need to.

I think the point is, a beginner coming across some code using such esoteric features is going to be a bit lost as to what's going on. With most of Python you can see what it's doing even if you didn't know the syntax before. That's true of basic f-strings, but not so much with some of this stuff.

Yeah languages are meant to be read even by beginners. I think a language like python shines by being small (in design) yet powerful (by capability). Having myriads of "nice for one use case" feature in the language is not the way forward. I long for the time where % format and .format are removed from the language spec. Even if they still work on the implementation for compatibility. Please remove things.

> Please remove things.

We're not done with Python 2 yet. This would be a Python 4 for sure. That's not according the philosophy of backward compatibility of Python (even if I agree with you).

Yes backward compatibility is not an option (lets avoid redoing python3 trauma ) and simply delegating coexistence of old and new features to implementation is a bit lazy. However documenting one way as the current one and other features as old ways, for backward compatibility can simplify things.

I don't get it. Why not just write:

>>> f'{foo}=' 'foo=5'

there's a pattern in debugging that basically looks like

   print('foo=',foo)

like all languages should, if a pattern is wide-spread, useful and basically boilerplate, python decided to create a language shortcut for it.

   print('f{foo=}')

f before the quote though

doh! classic 'your bick is dig', can't even edit now :(

As someone who probably relies on print statements too much for debugging, I love this.

Conda with pipenv = great

it's great if using debian where it's still on 3.5. installing miniconda lets you have the newest python easy-peasy

I just want to thank the author for not using Medium.

Far too many tech blog posts use that platform now and I don't like it very at all as it feels really bloated and I can never be sure if what I'm about to click is a 'premium' Medium post or not.

Glad to see I'm not the only one thinking like this. Medium is the MySpace of the late 2010s.

Good blog post by the way.

I like that the article doesn't have generic meme images separating each paragraph.

A few years ago complaining about that would get you flagged. Now it gets upvoted.

I talked about not using it years ago on my YouTube channel and got a rash of shit. I've shown how to replicate that site in minutes as well.

Wow, Medium built such a dysfunctional webservice that people start actively thanking others not to use it - in completely unrelated posts. Nice job.

Been enjoying the Make Medium Readable Again (https://chrome.google.com/webstore/detail/make-medium-readab...) plugin. Removes all the clutter from the page, you won't even know you are on Medium!

Alternatively if you have uBlock and are scared of one-off extensions, you can add this generic modal filter[0] list from the webannoyances repo.

I found this by searching "modal" on filterlists.com and clicking the subscribe button. IMO there should be a Medium-specific one like they have for stackoverflow or youtube.

I cannot personally vouch if the modal filters are overzealous. I'd rather avoid Medium than putting up with this sort of thing.

[0] https://raw.githubusercontent.com/yourduskquibbles/webannoya...

Better link: https://github.com/yourduskquibbles/webannoyances/

Thanks I should have included the repo.

I only found the "one-click" subscribe for the masterlist in the repo readme. Additionally I couldn't link a specific filterlists.com result to the modal filter "one-click" subscribe that I use.

I'd have preferred a ublock subscribe link but I'm not sure how to make it clickable without an href.

  abp:subscribe?location=https%3A%2F%2Fraw.githubusercontent.com%2Fyourduskquibbles%2Fwebannoyances%2Fmaster%2Ffilters%2Fmodal_filters.txt&amp;title=Web%20Annoyances%20Ultralist%20-%20Modal%20Overlay%20Filters

In case anyone is stuck on older versions of Python for any reason, here are some backports of some of the listed features:

* f-strings: https://github.com/asottile/future-fstrings (this is a pretty crazy hack, all the way to Python 2)

* pathlib: https://pypi.org/project/pathlib2/

* typing: https://pypi.org/project/typing/

* enum: https://pypi.org/project/enum34/ (doesn't include any additions in 3.5+)

* functools.lru_cache: https://github.com/jaraco/backports.functools_lru_cache

* dataclasses: https://github.com/ericvsmith/dataclasses (3.6 only, requires ordered class __annotations__)

I'm half-expecting someone to come along with a hacked-up Python 2 backport for implicit namespace packages :)

With that in mind, the end for Python 2 is less than 8 months away.

Something fun to help people feel the "impending doom" and port their projects:

https://pythonclock.org/

Python 2 is free software. Nobody and no proper subset of people can unilaterally end it. It ends when people stop using it, writing patches for it, and sharing those patches. There might be a lacuna of coordination when the current maintainers step down in 8 months, but the leftover stubborn Python 2 users will come up with some way to share their patches for, probably, decades.

I was using enum34... then one day I wanted to use a Flag and found it wasn't in there

the same author has another library: https://pypi.org/project/aenum/ which includes the latest enum stuff from 3.5 (plus some other things)

SimpleNamespace[0] is pretty handy too. It essentially lets you wrap a dictionary and access its keys as member properties.

Also, it's kind of weird that the article would mention type annotations and then not mention mypy[1].

[0] https://docs.python.org/3/library/types.html?highlight=types...

[1] http://mypy-lang.org/

Whoa, I've never seen the `types` module before! This will actually be very handy for me for what I've been working on lately. Thanks for pointing it out!

It's definitely worthwhile going over the standard library[0] every now and then. I always find something interesting to read up on. I recently learned about cmd[1] (which has been around for a long time) and have been using it to write a text adventure game.

[0] https://docs.python.org/3/library/index.html

[1] https://docs.python.org/3/library/cmd.html

Curious to see a small code sample illustrating typical usage of SimpleNamespace if you care to share? Docs didn't include one.

Here's one case I've used it for:

    foo = json.dumps({'a': 1, 'b': {'c': 2}}) # Pretend you've downloaded some JSON...
    bar = json.loads(foo, object_hook=lambda x: types.SimpleNamespace(**x))
    print(bar.a)
    print(bar.b.c)
    print(bar.b.d) # Error
    bar.b.d = 3
    print(bar.b.d) # Works now

I mean, you could pretty much do the same sort of thing with namedtuple, except this is mutable and doesn't require a pre-declaration of the keys. It's easier to write off the cuff.

Cool, thanks!

neat trick

Originally small dynamic languages like Python and Javascript are constantly growing new language/stdlib features but rarely taking anything away. I wonder how this affects learnability, how long will it take for a programming newbie to be able to jump into an existing project or read other people's code. I guess this could be measured and studied, I wonder if someone has done it.

I think Python was probably at a pretty good local sweet spot in the complexity:power ratio around the 2.0 version.

Python is definitely heading down that dark path.

I wonder how many pythonistas are fully across all of the new syntax coming, like positional-only args or even the infamous assignment expression.

I don't think it's the case that a language needs all of its syntactic sugar to be intuitive or obvious to the initiate. These are things you will acquire over time as you either search the internet for "how do I do X" or you read other people's code that exemplifies them.

Though I'd argue that anybody who adopts the "pythonista" label is likely to be somebody who looks through version release notes for these kinds of things. I certainly do!

2 new syntaxes a years is not a not. I find it quite balanced for a 25 years old language.

It's funny, because I always hear people complaining Python move too slowly to stay modern on one side, and then right after some other people saying it goes too fast.

My take on this is that people just like to complain.

Yes! Python is getting less pythonic with each new version.

i prefer to see it the glass half-full way: the definition of pythonic continuosly evolves.

Python has taken many things away, hence the 2 to 3 transition. And people didn't like it. Not one bit. We even had to bring features back (u'', %, etc) after taking them away.

Now I agree, if I could snap my gantlet I would remove a lot more. We don't need Template(), the wave module or @static and so many other things. And we could have cleaned up Python more deeply in the transition.

But reality is very messy.

Plus, you gotta realize that the Python community has nowhere near the resources of languages such as Javascript. JS is developed by giants that poured hundreds of millions in it and set dozens of geniuses to work solely on improving the implementations.

Python barely (and this is recent, we had less before!) has a few millions for operating the PSF related activities, which includes organizing PyCons all over the world and hosting python.org or pypi.org. Only a handful of people are actually paid to work on Python.

So you have a giant, diverse community, with billions of lines of code for so many different activities, and not enough resources to work on cleaning up everything.

Welcome to life, this stuff where if you want to get things done, you have to compromise.

In practice, at a workplace, you can probably remove quite a lot through linting or some other form on enforcement. In general, as long as the majority of people out there use the same good style and don't use stuff like Template(), I don't think it'll be an issue for newbies. The real issue is when you have many methods of doing the same thing that are all popular.

Yes but I do understand the problem. E.g learning packaging in python is a terrible experience, not because it's hard, it's really not, but because of the noise we accumulated.

> not because it's hard, it's really not, but because of the noise we accumulated.

In an effort to avoid the noise, would you be so kind as to point me in the direction of the current way to handle packaging in python? I don't work in Python consistently enough to be up to date on this, but every time I do it's hard for me to suss out the best way to package up and distribute the work for internal end users.

I don't know of any resource that make a good summary of everything, that is up to date and pragmatic.

The least worst is https://packaging.python.org/ but it skip major issues I know beginners have (multiple installed Python, path problems, etc), it's not clear what works on what OS. It also promotes pyproject.toml over setup.cfg, and ignore pex and nuikta.

That's one of the numerous short pocket books I should write.

Give me a book deal, I'll do it :)

Thanks for the link! I've worked with Python long enough to be familiar with those beginner issues, just infrequent enough (and even more so for stuff that needs packaged) that I've never quite been able to wade through the noise.

Also,

> not because it's hard, it's really not

and then

> Give me a book deal, I'll do it :)

gave me chuckle. Because they're both true: it's not necessarily hard, and yet despite that it still needs a short book to really tackle the subject. That's such an out of place state for something related to a core component of Python.

You still can remove from the spec of python 4 while letting cpython4 implementation keep using python3 constructs also.

brings to mind C++. If you learned it 15 years ago and haven't kept up, new C++ will look like a new language.

Don't think the same is true for python (yet?)

I'm training people in Python regularly and my classroom experience is: ALWAYS USE __INIT__.PY EVERYWHERE. Don't use this namespace feature. You think you understand how it works, but you don't. It will bite you.

I just read the docs on this, but don't really grok the pros-and-cons. Can you give an example of a gotcha? That would be very helpful.

Namespaces are useful for library providers and os packagers to split sub features of one package and distribute the chunks separately.

It will virtually merge dirs with the same name under some circonstances, and won't be importable in some others. You also lose the benefit of having an init file in which you explicitly setup the object you expose for import.

Amen. Much as I wish python importing was as powerful as js, that ship has sailed and we're nowhere close to getting rid of the dunder-init file.

Also pathlib is .. urgh. This is one of those that shouldn't be in the stdlib. I don't get why it is, but requests isn't.

Js has no import system. Bundlers do, and they are terrible at it though.

Modern js has native modules: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Which don't work in the browser, the main place we use JS.

It does work in all modern browsers except Edge. Which I grant you might be a sticking point. However, Edge is moving to Chromium in the near future.

Firefox 66:

import('/modules/mon-module.js') SyntaxError: dynamic module import is not implemented

So no network access, and of course no FS access.

Hence, doesn't work in the browser. You need a lib or a bundler.

I'm also using Firefox 66 and it works for me. Here's a live example: https://mdn.github.io/js-examples/modules/basic-modules/

Source code and other examples are at: https://github.com/mdn/js-examples/tree/master/modules

> Much as I wish python importing was as powerful as js

In what way is python importing less powerful than JS? My experience is very much the opposite (to the extent that JS even has an import system).

> I don't get why it is, but requests isn't.

Because pathlib was proposed (PEP 428), considered useful and self-contained.

The requests project does not want it to be included (this was discussed at length back in 2015), and it would require merging chardet and urllib3 into the stdlib first.

My post was unclear, it's fair that base JS doesn't have an import system; I was talking about ES6 imports.

And yes I'm aware why requests isn't in the stdlib, but that doesn't make it "correct". And chardet should definitely be in the stdlib. I mean, ffs, `mimetypes` is in the stdlib. urllib3 is more contentious but honestly? It probably should be as well.

What's your deal with pathlib?

For starters, I'd say that overloading the multiplication operator to concatenate strings and insert path separators is one very bad code smell.

We already have SQLite, difflib, et al. What's so bad about pathlib? I use it all the time, it's great.

pathlib is nice

Beware lru_cache and mutable objects. I'm on a phone so this will probably contain an error but:

    @lru_cache
    def slow():
        return {"result": "immutable?}

    answer = slow()
    answer["result"] = "no"
    print(slow())

Can't edit now, but here's a runnable gist https://gist.github.com/IanCal/d703106344e876728d2aaef67800d...

It also won't let you cache on mutable values, so you can't pass in lists or dicts. I've used it on occasion, a lot of the time I ended up rolling my own instead. Usually without expiration since I didn't need it.

The "Implicit namespace packages" item is not correct. Those are for a a specific packaging scenario which doesn't come up in normal usage. Packages should have an __init__.py file in general.

If the file is going to be empty I fail to justify the effort. It is implicitly a package. Is there a case where an empty init file makes sense in Python 3?

Mypy uses the init files to resolve absolute imports.

Its absence distinguishes a non-package from a package.

Thanks, this was the only item I wasn't sure about, and reading PEP 420 didn't really make it clear how this would benefit regular applications.

The example of f-string in the article is quite unfortunate, it suggests using it for logging messages. F-strings shouldn't be used in loggers, because they aren't lazy evaluated.

What instead?

Standard logging functions, without f-strings. So instead of: logging.debug(f'Problem with {x}') use: logging.debug('Problem with %s', x).

TBF for the vast majority of logging it doesn't really matter. Not to mention the laziness is only the (usually cheap) formatting itself. If you're logging the result of non-trivial expressions (which would otherwise not be computed at all) you have to handroll it.

f-strings are mostly unusable for translatable contexts. And format-strings also don't work then (they're a security issue).

I already experienced performance issue, just because log messages (millions/s) were evaluated, for logger in debug mode, so it really depends on the system.

A cookie for this man!

Wow, today I learned about f strings. That will make my code so much more readable! And no more annoying bugs because I skipped one of the seven references in the string!

It reads like sarcasm, but then you drop a real benefit. ((Confusion by New Order playing in the background))

No it’s true enthusiasm. The internet has just ruined ! For everyone. :)

Glad to hear you like it. As a fun fact (and self-back patting), it was I who braved the gauntlet at the python-ideas mailing list to get the idea accepted. Eric Smith did everything else so deserves the credit.

Background: turns out that the string prefix was the only ASCII syntax left in Python3, that's why it was chosen. The name "eff string" makes me cringe, but once it stuck there was no stopping it.

> it was I who braved the gauntlet at the python-ideas mailing list to get the idea accepted

That explains why you're all over this post ;)

Fantastic feature, though. Really. Thanks!

Out of curiosity, what name would you have preferred over "f-strings"? "Format strings"? "Interpolation strings"? "I-will-evaluate-expressions-in-curly-braces strings"?

Well, I usually participate in Python threads. This post had only one part on f-strings.

What to call it? Anything I guess. String interpolation is what it is called elsewhere. But the f prefix is what everyone sees, though as mentioned, it is only an implementation detail because other forms of syntax were already spoken for. I personally chose f for "format."

Early on Guido changed the scope to include expressions as well. People were already starting to say "eff string" so I changed my proposal to "e-string" for "expression." I also like the sound of it better, sounds like email, etc. But Guido (and Eric) decided to stick with f. Maybe because Guido learned English later he doesn't realize how unfortunate it sounds. But, years later we use the "iPad" and forget it sounds like a feminine napkin. So, no big deal in the end.

Don't forget @singledispatch! Every once and a while this tool is glorious for a polymorphic handler. Ie, `process_entity` that will fan-out to `process_movie` or `process_tv_show` etc...

I miss it in a Python 2.7 project I am on right now.

Link to the docs: https://docs.python.org/3/library/functools.html#functools.s...

It's the same concept as clojure's defmethod.

>f-strings

What happened to "explicit is better than implicit"? "Only one way to do stuff"? One of the more baffling Python features if you ask me.

This debate already took place. Go back to the python-idea mailing list posts about f-string for a complete back and forth on your point.

It took place three times actually; Ping proposed string interpolation in 2000 in PEP 215 https://www.python.org/dev/peps/pep-0215/ and it was rejected, or rather simplified by Barry Warsaw in 2002 in PEP 292, which was actually implemented: https://www.python.org/dev/peps/pep-0292/ and remains in modern Python. Barry's optimism that this would satisfy the demand for template strings was of course foolish, but it met with Guido's misguided approval at the time. Fifteen years later, it was proposed a third time as PEP 498 and finally got in in a sane form: https://www.python.org/dev/peps/pep-0498/

Exactly, the f-string syntax looks so close to PHP’s way of handling strings that I’m really amazed that it managed to pass and be implemented. I mean, I don’t see that much of a difference between

> name = ‘Jane’

> print(f”Hello, {name}”)

and PHP’s:

> $name = “Jane”;

> echo “Hello, $name”;

Why is looking similar to the PHP method inherently a bad thing?

About 15 years ago when I was starting my life as a professional programmer PHP's way of handling strings was seen as inferior to stuff like:

> name = 'Jane'

> print 'Hello, %s' % (name)

but I can't exactly reproduce all the discussions from back then. As for me personally I can see that I dislike the new

> print(f'Hello {name}')

format because my editor cannot instantly show me that {name} is in a fact a variable, and now that I wrote this down I remember that this was one of the complaints made against PHP's use of "Hello, $name" . But maybe newer editors do in fact recognize {name} as a Python variable, in any case it looks and feels "impure", for a lack of a better word.

VScode and PyCharm recognise variables in f-strings correctly. I'm not sure how "%s" is more pure than "{name}". They are both variables, only in the case of "%s" you add a level of indirection.

PHP has many problems, but I always considered string handling one of PHP's strengths. Considering how PHP's string interpolation has gone mainstream I don't seem to be the only one:

PHP:

    print("Hello, $name")
    print("Hello, ${getName()})

Python:

    print(f"Hello, {name}")

    Console.WriteLine($"Hello, {name}")

Javascript

    Console.Log(`Hello, ${name}`)

Ruby

    puts "Hello, #{name}"

etc.

Is that really PHP's, though? It's been in shell scripts and perl for quite a while...

That's an interesting question. I couldn't find any summary of the history of string interpolation. The first version of PHP in '94 had it. Bash predates that by 5 years and Perl by another 2, but I'm not sure about the initial feature set of either of those. The first POSIX standard for command interpreters is from 1992. I suspect the feature comes from early versions of csh in the early 80s.

However I think it's fair to credit PHP with introducing generations of programers to the idea.

Bourne shells as early as Version 7 Unix (1979, [1]) had rather elaborate interpolation of variables into strings (i.e., not just ${foo} but ${foo?bar}, etc.).

I'm reasonably sure that earlier versions of sh had some facility for this as well.

So, it's not really a PHP innovation, although some people might have seen it first there.

[1] https://www.in-ulm.de/~mascheck/bourne/v7/ -> "Parameter substitution"

Apparently Unix Version 6 (1975) used Thompson Shell which (provided the manuals are correct [1][2]) didn't support arbitrary variables (and even environment variables were not a thing back then). However for use in scripts $0, $1, etc were replaces with the script name and the arguments.

1: https://www.in-ulm.de/~mascheck/bourne/v6/

2: https://etsh.nl/man/_tsh.1.html

String interpolation was there in the first versions of the Bourne shell, and I suspect the pre-Bourne shell in PDP-7 Unix in 1970 or so, but of course programmers have been generating strings by pasting together bits of constant and variable strings for a lot longer than that; Church defined the λ-calculus in terms of string interpolation in the 1930s, before there were even any computers. Computer-generated printed letters were already a commonplace occurrence in the US in the 1950s (at least if references in the fiction of the era are to be believed), and those are of course built with string interpolation. Around that time, assembly languages became the first languages to claim to make computer programming obsolete (because at the time "programming" meant programming in binary), and rather quickly some of them developed macro capabilities — before 1960, I think. Macro assemblers typically do their macro magic with string interpolation, even today. High-level languages of the time commonly used template strings, like those in (1964?) BASIC's PRINT USING statement, though FORTRAN's "edit descriptors" used a non-interpolation-based syntax. COBOL's "edited picture" feature is a kind of string interpolation, but I have no idea when it was added to COBOL.

In the 1970s string interpolation really got going with languages like sh, cpp, and m4, which was popularized on many platforms by the "Software Tools" book (which in a sense is a slow buildup to the presentation of m4); m4 is Turing-complete entirely through string interpolation (more so even than Tcl decades later).

I suspect Perl had string interpolation from the beginning (1988), given its shell roots, but I don't have a copy of Perl 1 to test with.

For an overview of modern languages' string-interpolation syntax, I suggest https://www.rosettacode.org/wiki/String_interpolation_%28inc....

Hope this helps!

f-strings are not the same way to do things but a different way. You can't explode a dictionary into an f-string to fill in the fields, for example, but you can in .format().

You seem to have forgotten "practicality beats purity"

f-strings is the right way to do this going forward, the rest will die off.

Another useful feature of 3.7+ (also available as a backport by the same name) I’ve not seen mentioned is: ContextVars

The gist is you get a single interface for any type of threadlocal storage that also works with coroutines (asyncio). I use it pretty consistently for context logging of long running processes in our systems. However it has implications on other concurrency paradigms usages as well.

https://docs.python.org/3/library/contextvars.html

1. f-string and extended-iterable-unpacking reminds ruby. i was really missing them.

2. pathlib is cool. will come pretty handy.

3. Data classes sounds good but there should be only one way to do things, even you end up writing few more lines.

4. type-hints are ok for very very big projects.

5. No comment on implicit-namespace-packages yet as still trying to understand solid use-case.

> Data classes sounds good but there should be only one way to do things

Dataclasses greatly simplify generation of simple classes which are unnecessarily filled with boilerplate. I'd argue that they are the one way to do this, and that choosing to manually implement the same naive `__init__` function for every simple class you write is the "wrong" way.

(Here I take "simple class" to mean a class which takes in some values during initialization and stores them without anything terribly interesting going on during that initialization. A simple class can have methods defined on it, though.)

> type-hints are ok for very very big projects.

> No comment on implicit-namespace-packages yet as still trying to understand solid use-case.

I think it's just... trying to keep your directories "clean"? I also don't really understand the use of this haha.

> Dataclasses greatly simplify generation of simple classes which are unnecessarily filled with boilerplate

I would love to solve these boilerplate kinds of problems using plugging like emmets. Magic like this feels good when writing but troubles during debugging and customizing.

Additionally naive programmers always use such features in wrong situation if they don't get use-case. I really feel sad after seeing so much bad-django-code in production.

> 4. type-hints are ok for very very big projects.

Great for smaller ones too! I type annotate every function, even

    def parse_args() -> argparse.Namespace:
        pass

at the top of simple scripts. With a good IDE (VS Code fits here too), the extra language lookups/insight you get are awesome. Command+mouse_hover gives me a great bit of information about args passed into the function if I annotate the inputs.

I tried to use type hints but I do not see enough value in them to justify making code more complex. It looks to me that they are similar to hungarian notation as they make refactoring harder but they are not reliable and does provide little value in checking program correctness. They are just documentation embedded into variable declaration used only for linting so I tend to not trust them.

Why do you say they are not reliable? Are you unfamiliar with mypy?

Because it is not used by runtime. Mypy is just linter which adds additional restrictions on code which are not enforced by interpreter. You can have false positives so correct python code must be fixed to pass linter checks. In my opinion if you need static type checks you should use statically typed language and not hurt your coding speed by such partial solution. Also majority of python libraries do not use type annotations so you are limited to your code.

Well, yes, this is exactly how the type checkers of statically-typed languages work: as linters, during compilation. The largest difference is that typing in Python is gradual, so not everything needs to be typed as you say.

I agree that statically typed languages are preferable. I don't consider static typing optional so I'm glad Python is growing a solution that supports it, though.

In my experience, mypy works surprisingly well. It doesn't hurt my coding speed at all but enhances it to levels that were not previously possible in Python, due to a lack of typing. False positives are rather rare.

don't use them if you don't validate them with mypy or similar.

> there should be only one way to do things, even you end up writing few more lines.

the zen of python never mentioned that you should exclusively code in machine code.

importlib.resources is an excellent replacement for pkg_resources.

with importlib.resources.open_binary('my.package', 'foo.png') as f: ...

Avoids the overhead of importing pkg_resources, and hassle of having to call cleanup_resources before the interpreter shuts down. Backports are available for Python <= 3.6 too!

I'm by no means a hardcore python dev but I feel like these are all pretty basic, common things. No?

I don’t think an LRU cache is basic

If you’ve ever needed it, and I imagine most people have, it shows up on any web search.

That being said I usually end up rolling my own that can serialize complex objects.

If you’ve ever needed it, and I imagine most people have, it shows up on any web search.

It didn't last time I googled it (because it didn't exist then), so I had no idea it existed.

They're basic but also things you'd never look up, especially if you've been doing python a long time.

For example, I just learned about f strings, because why would I need to ever look up if there is a replacement for .format()?

And the LRU cache. I've been hand rolling that for years, but I never thought to see if they had added it to the standard library, because why would I look?

It pays to look thru the stdlib every so often, there is just so much there, and plenty of dark corners (in the Warcraft sense) to look in.

I hand rolled stuff in itertools/functools for years before discovering them.... collections Counter, etc.

Or how about str.partition instead of try: str.split() ?

On a related note, every couple of years I make an effort to man page every cli tool I use, and re-read the man page, just in case. Most times I discover there is some argument that I've missed before in a tool (or has been added in the intervening time)

When my dad sold cars he’d always tell his customers to read the car manual again after six months for the same reason.

don't you read about what's new when new versions come out?

I've been programming python, on and off, since 2001, and I still learnt a couple of things.

TIL - Enum, auto(), @dataclass, and that classes don't need (object) anymore. Neat!

EDIT: nope, nope, this is all wrong. My mistake. Ignore everything else here.

> classes don't need (object) anymore

My friend, old-style classes became unnecessary with the release of Python 2.2 — almost 20 years ago now [0]. I'm afraid you've been living in the past. :)

[0] https://wiki.python.org/moin/NewClassVsClassicClass

Nope, she/he has not. (object) were the new style classes!

Oh man, major brain fart there. You're completely right of course. I was thinking of the transition to classes-as-types in general.

Probably would've helped if I actually read the page I linked...

Extending `object` was necessary until 2.7 when it became the default and old style classes were removed entirely in 3. So I think while the GP comment recently learned they don’t need to extend `object` anymore, they are/were familiar with new style vs old style classes.

Right! Yes, another commenter pointed that out to me too. I made a goof and misremembered some things haha. Oops.

New-style classes never became the default in Python 2:

    Python 2.7.15 (default, Jun 27 2018, 13:05:28) 
    [GCC 8.1.1 20180531] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> class Foo:
    ...   pass
    ... 
    >>> Foo
    <class __main__.Foo at 0x6218e384ee88>
    >>> class Bar(object):
    ...   pass
    ... 
    >>> Bar
    <class '__main__.Bar'>
    >>> isinstance(Bar, type)
    True
    >>> isinstance(Foo, type)
    False

Thank you for the correction! I’ve not used Py2 in 6 or so years.

A funky thing about the Enum class is you may also want the `@enum.unique` decorator in most cases, which enforces the enum/value pair being unique. I found the documentation a little misleading until I found that [0].

[0] https://docs.python.org/3/library/enum.html#enum.unique

I'm not sure I follow the purpose of this, and the documentation is not terribly elucidating haha.

What's a case where I'll have a problem if I don't use this decorator? (I've used the Enum class a bit since it became available, but not in any particularly interesting implementations so maybe that's why I've never heard of this.)

``@enum.unique`` will raise a ValueError if you accidentally include the same value twice in an enum definition:

    In [21]: @enum.unique 
        ...: class Animal(enum.Enum): 
        ...:     DOG = 1 
        ...:     CAT = 2 
        ...:     PIG = 2 
        ...:                                                                                                                                                              
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-21-cbc1625bb41c> in <module>
        1 @enum.unique
    ----> 2 class Animal(enum.Enum):
        3     DOG = 1
        4     CAT = 2
        5     PIG = 2

    ~/.virtualenvs/py3/lib/python3.6/enum.py in unique(enumeration)
        834                 ["%s -> %s" % (alias, name) for (alias, name) in duplicates])
        835         raise ValueError('duplicate values found in %r: %s' %
    --> 836                 (enumeration, alias_details))
        837     return enumeration
        838 

    ValueError: duplicate values found in <enum 'Animal'>: PIG -> CAT

It's intended to catch programmer errors - not really useful for small enums like this one, but in a large enum, having duplicates could be difficult to notice, and might cause some insidious bugs.

Oh, I get it! This makes sense.

So this is mitigated by using `enum.auto` for setting the values. But I can definitely see how `enum.unique` would be useful if your enums had more specific values.

Thanks for the explanation!

> What's a case where I'll have a problem if I don't use this decorator?

If you are explicitly setting enum values, it causes accidental aliases (duplicate values) to be an error early rather than a source of (potentially subtle) problems later.

It also specifically communicates intent to other readers of the code, which can be just as important if your code has a lot of enums,some of which do use aliasing intentionally.

I see, okay, this all makes sense. Thanks for the concise explanation!

You can now use Python Scrapy with the latest versions of Python 3 for web scraping. Here's an example https://youtu.be/qj6H38Xwvu0

It even works on Windows.

I would add to this list:

  * generators
  * defaultdict
  * generators
  * sets

Generators and sets have been around for a long time I think.

They were there in 2.X version like 9-10 years ago when I first used Python anyway and aren't 3 specific.

Generators were added in Python 2.2: https://docs.python.org/2/whatsnew/2.2.html#pep-255-simple-g...

Built-in set objects were added in Python 2.4: https://docs.python.org/2/whatsnew/2.4.html#pep-218-built-in...

defaultdict was added in Python 2.5: https://docs.python.org/2/library/collections.html#collectio...

I was curious to see how asyncio has matured into such a list.. but it was missing the list. Is it common enough for people to use in scripts to speed up concurrent work yet?

Anecdotally, no. From where I sit, the multiprocessing and threading modules still seem to be the main way to to concurrency in python.

From where I sit (I've been soloing python projects for a few months now) For IO-blocking tasks, asyncio is the way to go and threading is very rarely needed if any. It is very simple to place a bunch of tasks into a list and do asyncio.gather on them, or add timeouts to async methods.

For CPU intensive tasks, asyncio actually provides it's own syntax for process pools which are easily interchangeable with thread pools (just change an import alias)

We actually tried switching to asyncio in production, but got worse cpu utilisation and worse latency (vs greenlet). After some benchmarking and tuning, we got a bit better latency, but CPU utilisation was still worse, so we actually ended up swithcing back to just using greenlet :(

I would not recommend using asyncio for cpu-intensive tasks. Processing pools are much better for that.

Is type hinting being adopted by Pythonistas today? I've been aware of the feature but haven't used it. To be a modern Python programmer, should I start using it?

I use it everywhere because it vastly improves code readability/usability in my IDE (PyCharm). But I don't know if I'm representative of the general Python population.

Not sure anyone will see this but this article is partly trash. When your post is titled "things you are not using in python3 but probably should" and the first section is on f-strings... I was tempted to close the tab then , but i followed thru and did find some interesting things.

But seriously f-strings? people have been talking about these for years now, it's not new is it?

If you look through the other comments here, you'll find plenty of people who didn't know about f-strings prior to this article.

f-strings were added in Python 3.6, released in December 2016.

All the other features mentioned in the article, except data classes, are older than that.

I didn't start using python til 3, but list comprehensions and f-strings are great. I've really been enjoying the coding style.

Here is another short guide on many useful features of Python 3: https://github.com/arogozhnikov/python3_with_pleasure

I really hate that __init__.py thing about modules it confuses the F out of teachers and students. Using files without a "program" on the command line is a massive leap as it is.

I read quite some things about the data class... in short: If I use a lot of pandas.DataFrames, is this something I should learn about?

No. Data classes are a mutable replacement for collections.namedtuple.

Thanx!

How do you define class variable in a data class?

You can use `typing.ClassVar[<datatype>]`

typing.ClassVar. See https://docs.python.org/3/library/dataclasses.html#class-var...

Thanks for all these.

There's no reason why Python 2 can't have the vast majority of these features. It's a shame that the Python continues to ignore the reality of the mess they made with Python 3, and failing to question what was an originally bad idea.

This question is done and dusted. There's literally no point in discussing it any more. CPython 2.7 support ends in 7 months. It's time to move on.

> There's literally no point in discussing it any more. CPython 2.7 support ends in 7 months. It's time to move on.

Ah, the classic "nothing to see here, keep moving on" argument. The lack of admission of a bad idea is itself a problem. What happens in a few years when Python 4 renders all Python 3 code incompatible?

> What happens in a few years when Python 4 renders all Python 3 code incompatible?

Ah, the classic "they did it once so they'll definitely do it every single time from now on" argument.

Nick Coghlan, one of the Python core developers, said "Python 4.0 will merely be "the release that comes after Python 3.9"" [0], so I think your concerns are ill-based.

[0] https://www.curiousefficiency.org/posts/2014/08/python-4000....

Not to mention some of us are glad lots of things got fixed and not sore in the slightest.

Oh, 100%. I'm right there with you.

> Not to mention some of us are glad lots of things got fixed and not sore in the slightest.

The people who are not "sore in the slightest" probably don't have to deal with large Python 2 codebases that rely on libraries that have no plan to move to Python 3. They are typically people who are writing code from scratch.

Rendering a huge codebase obsolete in order to upgrade print statements, import statements, and internationalization (i.e., unicode) was not a fair trade for many existing projects.

Been writing Python since roughly 2000.

My stuff was not overly concerned with encodings, that part was luck. I moved to the logging module early, avoiding print problems, that was smart. Other changes were trivial mechanical fixes, that was easy.

For the unlucky, obtuse, or resource challenged, you've had an extra ten years of support. That's sufficient IMHO. It is time to suck it up and port, or retire the app. Constant bitching just gets old.

Meanwhile, I'm using a great language that will be relevant for years.

> one of the Python core developers, said "Python 4.0 will merely be "the release that comes after Python 3.9"" [0], so I think your concerns are ill-based.

Why would they say that? They know breaking compatibility was a huge blunder, but they ignore the folks that it affected. Python 3 promoters are typically folks that don't have to deal with mess it created. If the Python foundation actually admitted to their mistake, it would engender far more confidence and credibility than pretending it does not exist.

No, there's no reason Python 2 can't have these features, but there's no particularly compelling reason to invest the engineering effort to develop it.

> the mess they made with Python 3

What mess, exactly?

> what was an originally bad idea.

What makes Python 3 a "bad idea"?

> What makes Python 3 a "bad idea"?

Breaking compatibility with a huge code-base for incremental features.

The move to Unicode is not "incremental" by, like, any measure. And other things, like moving `print` to be a function instead of a statement, were absolutely good design choices and worth breaking for.

People have had plenty of time to work on porting their code from 2.x to 3.x. Honestly, I think it's remarkable the devs kept maintaining 2.x as long as they did. I mean, look at Swift, where they had breaking changes year after year, and yet their user base has fairly exploded since the language's release not so long ago.

Just... get on with it, you know? Move to Python 3.x and be done with it. There's no particularly good reason to stick to 2.x forever.

Unicode alone is worth moving.

I'm not particularly excited about a lot of other new features though.

I don't see a real reason for type annotations from my perspective (if I wanted typed I'd use a typed language, if I wanted types indicated I'd indicate in the doc strings).

Also f-strings which I haven't used it. They probably are better, but I'm barely getting used to `format` as opposed to `%s` (which admittedly `%s` does suck as I've said many time while counting items on the screen with a pencil).

> I don't see a real reason for type annotations from my perspective (if I wanted typed I'd use a typed language, if I wanted types indicated I'd indicate in the doc strings).

I definitely get this perspective, but I actually use type annotations a lot. I do the vast majority of my Python development in PyCharm, and using type annotations greatly facilitates auto-complete features. It also helps me do some very simple debugging while writing, as the IDE will tell me "Hey! I don't think these types match up!", which is often enough to save me the headache of debugging.

Realistically, I just want Python but with a static type system. Which is why I'm implementing that as my own project haha.

> Also f-strings which I haven't used it.

Oh, dude, you gotta get on board with f-strings. They're soooo much simpler than the alternatives. Just skip learning `.format` and go straight to f-strings. I moved to f-strings when they first became available a few years ago and haven't looked back since.

Dataclasses are also great, IMO. I prefer algebraic data types in functional languages like Haskell or OCaml, but I think Python's dataclasses are a "good enough" solution for me to use in the meantime. I often define very simple data types and hate the default `__repr__` implementations and writing naive `__init__` functions. Dataclasses make all this much faster and, I think, more readable.

Don't need to skip learning format. f-strings are a terser version of the .format call with a bit of superset features thrown in.

Ah, yes, I just meant they shouldn't focus on using `.format` and should instead just use f-strings directly. Didn't mean to imply that they were totally separate things, though that's definitely how my phrasing came across.

There's no question that Python 3 is an improvement, but the question is whether it's an improvement worth breaking every existing Python app. Especially when many of Python 3's features could easily have been backported to Python 2. Python 3 is effectively a new language. People will be running Python 2 for many more years to come.

format has been around for a decade, and you're just getting used to it?

Yep, the other way is easier (if, and only if it's short), it has simpler rules, less typing, and that's how I learned it and it's in muscle memory without having to stop and think or look at examples.

> The move to Unicode is not "incremental" by, like, any measure.

The unicode improvements in Python 3 are definitely a step in the right direction. But there were certainly good ways of improving unicode issues without breaking compatibility. Things like print are a joke, no one should care about its semantics.

> Just... get on with it, you know? Move to Python 3.x and be done with it.

Pretty easy to say. But try convincing product managers to take 5 devs off their current projects for 6 months to port a working production app to a newer programming language, with little concrete evidence that it will have a meaningful impact on the project (other than introducing bugs).

Guido had a good retrospective on the shift, while the change was hard and painful, it needed to happen for the language to be easier to work with longer term.

If you're interested it's worth a watch: https://www.youtube.com/watch?v=Oiw23yfqQy8

> Guido had a good retrospective on the shift, while the change was hard and painful, it needed to happen for the language to be easier to work with longer term.

He was wrong. It is far harder than he anticipated, and it did not "need" to happen to grow the language.

Can you expand on why/where Guido was wrong beyond why he cites where he (and the core team) made mistakes in the linked talk? I'd disagree about the "need", as the fundamental changes (meaning non-package renaming changes) were pretty important for the increasingly international economy we are in.

There is a reason to get rid of bloat. It makes for a more orthogonal language. You don't have randomly sprinkled syntax for printing characters, raising exceptions, catching exceptions, handling strings, implicit type conversions when comparing values, etc which were bad ideas and good riddance.

Reply on news.ycombinator.com