Altering Python attribute handling for modules | Hacker News Reader

Altering Python attribute handling for modules

lwn.net

50 pointsjwilk23 comments

I always wondered why the import syntax is so different for the two things:

  import mpmath as mp
  from mpmath import mp

IMO that's the root cause of that kind of user errors. Why not just

  import mpmath.mp as mp

or something that has basically the same syntax for importing an object as for importing the module?

> Why not just `import mpmath.mp as mp` or something that has basically the same syntax for importing an object as for importing the module?

Because if we applied that as a "rule", then an import like this:

  from keras.applications.vgg16 import VGG16, preprocess_input

Would become

  import keras.applications.vgg16.VGG16 as VGG16
  import keras.applications.vgg16.preprocess_input as preprocess_input

Which seems unnecessarily long.

Instead, I personally don't understand why everyone shortens numpy to np. I suspect it's because it's mostly used by people with a math background, who also believe that greek letters make for better variable names than actual spelled out variable names.

It would become

  import keras.applications.vgg16.VGG16
  import keras.applications.vgg16.preprocess_input

Which is still long but more explicit (and python likes that). Imports cannot have dots so by default it should use the same name as the object itself, which is the word after the last dot.

If still necessary to differentiate, the perfect alternative would be to flip it.

  import VGG16, preprocess_input from keras.applications.vgg16

In fact I've always find it strange that an import needs different syntax depending on how you use it. If all imports started with "import" it would be much more consistent and easy to read.

Python imports don't work like in Java.

  import keras.applications.vgg16.VGG16
  import keras.applications.vgg16.preprocess_input

  preprocess_input(...)  # ERROR
  keras.applications.vgg16.preprocess_input(...)  # CORRECT

  import VGG16, preprocess_input from keras.applications.vgg16

This approach has a couple of downsides:

- no editor autocomplete while typing (JS modules have the same downside)

- more difficult to read multiple imports from modules in the same hierarchy (since the "from" parts aren't aligned)

- import sorting becomes less clear (do you sort by imported things or by the module you import from?)

JS uses your suggested format, and I dislike it much more than the Python equivalent due to the reasons stated above.

I agree, both those libraries just have bad api conventions leading to user errors. They've just become so pervasive now that bad habits have become the standard convention in those spaces.

In Clojure we have (using :require in the namespace declaration):

(:require [foo.bar])

(:require [foo.bar :as quux])

(:require [foo.bar :refer [x y z]])

…etc

There’s no reason you couldn’t have similar syntax in Python.

I dislike the renaming of imports unless it's something that needs to be done for specific reasons, like two different conflicting names or doing something sneaky. At that point though, it's still generally possible to work around it with more qualified names.

It's something of a lost cause in the numpy/pandas ecosystem though, since every single example has them renamed this way and it induces just the cognitive load that I'm trying to avoid.

Franky, it's because python is only barely tolerable vs actual array-centered languages like Fortran or Matlab or Julia. It's bad enough having to sprinkle "np." and extra parentheses of various flavors and dangling commas everywhere along with whitespace eating up columns. The comparison should be with Matlab or Julia. Most people writing numpy-based code are trying to replace Matlab and Fortran.

Maybe you could be pedantically happy if "numpy" itself were renamed "np"? Then we can just "import np" and spare you from a DSL nightmare and ourselves from a few keystrokes while you can move on to complaining about obtuse, ungoogleable and inscrutable package names instead.

No, that's not it, that's just a thing that happens.

I'm thinking more of a community that can't decide between

  from foo import toolkit
  from foo import toolkit as tk
  from foo.toolkit import thing1, thing2
  from not.foo_toolkit import internalThing1, internalThing2

and the pain it is to deal with reading code there.

I thought you were going to suggest:

    import mp from mpmath

I do dislike that the order changes, even though it is consistent in terms of the 'path', because it means everyone groups by keyword, which then puts paths out of order/same ones separate anyway.

also means no autocomplete with the from bit first

    import mpmath.mp as mp

Doesn't work in this case because mp isn't a submodule of mpmath. You will get "ModuleNotFoundError: No module named 'mpmath.mp'"

The "from" is what dives into a module and takes something out. It's not just defining a syntactic macro. You would have to do something closer to this:

    import mpmath
    mp = mpmath.mp
    del mpmath

to replace what "from" does.

That's an implementation-detail. Nothing prevents the import from diving automatically into the file. But doing so would open the door for name-shadowing, as with a package you could have a separate mp.py for import, or a definition of mp in __init__.py. And this would be another confusion for people, and a performance-problem maybe.

I get where you're coming from, but those lines aren't equivalent (which may be why its confusing). "as" changes the name and can be used for both:

    import mpmath as mpmath_module
    from mpmath import mp as mp_context

"import" and "from" always address modules (directories and files) from/import extracts objects from modules and "as" swaps out the name.

It's confusing because "from/import" reaches inside a package and Python uses the same delimiter for Namespaces/Modules, Classes, and Properties.

urllib.request.HTTPBasicAuthHandler.auth_header is Namespace.Module.Class.Property You have to use each one differently...but you just have to know. You get some small context clues based on capitalization or knowing the first thing at the top of the file is a Namespace or Module.

In the very typical for Python community way, an effort was made to put more training wheels on a bicycle that already has like six pairs of them.

1. This is not a real problem (this only affects completely inexperienced users for whom everything is a problem anyways).

2. The real reason for a problem is the existence of multiple syntactical forms for import. To solve the problem one would have to remove the unnecessary import forms (but, come on, there isn't really a problem, so nothing needs to be done).

3. Modules historically struggled to be "real" objects in Python. So, another direction this could've taken is giving up on making them real objects. Just make them a syntactic convention (like packages in Java). But, again, there's no real problem here, so nothing needs to be done.

> this only affects completely inexperienced users

I'm not sure that's true. It doesn't seem to me that all of the users who encounter the mpmath issue described in the article are inexperienced users. They just aren't experts on the internals of Python, nor should they need to be.

> The real reason for a problem is the existence of multiple syntactical forms for import.

I don't see how removing the "unnecessary" forms (I don't agree that they're unnecessary, but I'm putting that aside here for the sake of argument) would help. Can you elaborate?

> giving up on making them real objects

PEP 562 already moved in the opposite direction, towards making them real objects (or more precisely, towards exposing more of their real object interface at the Python level).

You need to be at the level where you understand the difference between import x and from x import. This comes after couple months of training. And if after that you accidentally mistype it -- well, you'll know where to look for a potential error. It is not a problem for anyone with significant Python experience.

> "unnecessary" forms

The from x import is unnecessary and so is import as. Both can be written like so:

    import x.y
    y = x.y  # from

    import x
    y = x  # as

So, they aren't necessary. You can write any program that uses them without them.

> PEP 562 already moved in the opposite direction, towards making them real objects

Who cares? We had dozens of nonsense PEPs, and will have hundreds more. Python isn't known for good planning nor for execution. I'm talking about what should've been done. The nonsense Python does is way over there when it comes to what it should be doing.

Python is a common choice for "first programming language." Whats the problem with making it more friendly/helpful to beginners?

I'm more worried that this PEP will make things more confusing for beginners. I see this PEP as a removal of some training wheels. It opens the door for much weirder behavior than warnings/error messages. Whenever I've reached for this feature, it's been to do something "clever" that I probably shouldn't have been doing in the first place. Sure, some people will do wonderful/helpful things with this feature. But lunatics like me will use this to do some absolutely unhinged things.

I hope this PEP gets accepted, but I understand if the maintainers decide it's a bridge too far.

That's not making it friendly for beginners.

What this PEP suggests is yet another clutch for advanced users. It will allow advanced users to screw with module properties in all sorts of unpredictable ways. Beginners will be just a little bit more screwed if this feature is added. But they are already in such a deep hole if they started with Python... I don't know how to help them except by suggesting they use a different language.

There are more inexperienced users than experienced ones, so if it really does disproportionately affect inexperienced users, it’s an even bigger problem!

I think the point is that this issue is occurring because of bad conventions and api designs that are pervasive in one area of the python community. It seems a bit like the tail wagging the dog to change the design and implementation of the language for the singular purpose of guarding against confusing import statements that most of the community would recommend not to use in the first place.

> 1. This is not a real problem (this only affects completely inexperienced users for whom everything is a problem anyways).

To be fair, mostly nothing in python is a "real problem", it's all just syntactic sugar to make coding more comfortable.

> 3. Modules historically struggled to be "real" objects in Python. So, another direction this could've taken is giving up on making them real objects.

No, go the opposite. Just make them first class objects. There is valuable comfort in this. Python is multi-paradigm anyway.

Reply on news.ycombinator.com