Hacker News Reader

Comment by captainmuon | Hacker News Reader

There are two things that are annoying about Python's import system.

Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...).

Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. Virtualenvs are the canonical way to do it, but to me it feels like a hack - the core language seems to wants all packages installed in /usr. So now I have all these virtualenvs lying around and they are detached from the scripts.

Why couldn't the import system resolve versions, too? You could say `import foo >= 1.0` and it would download it to some global cache, and the import the correct versions from the cache.

View on HN

Once I wrapped my head around when you can and can't use relative imports, I've been pretty ok with them. The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`.

Coming from JS, that was a pretty frustrating realization.

> The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`.

This.

IMO that's also one of the main issues of Bash – you can't modularize a script unless you make sure your working directory is the directory containing the script. (And good luck with finding out the latter! [0])

[0]: https://stackoverflow.com/questions/4774054/reliable-way-for...

I actually spent time wrestling with exactly this earlier today, and I think I finally have a 2 line solution for it:

https://pastebin.com/5WCXb2Gg

Assuming `foo.bar` is a script inside the package `foo` that you want to both import and run directly (without `python -m ...`), this lets you do so without too much hassle.

> The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`.

It does not though, unless you have altered the default sys.path to always contains `.`.

When running a file / script, Python will add that file's directory as the first lookup path, so these two invocations should have the exact same sys.path, and thus the same module resolution throughout[0].

If you have added `.` to sys.path (which is not the default), then the first invocation will also have the "grand-parent" folder on the sys.path, while the second won't.

This also doesn't seem to have anything to do with relative imports, you need to already be inside a package for relative imports to do anything.

[0] that is not the case of `-c` and `-m`, both add CWD to the path, and they differ in what they do: `-c` adds `''` to sys.path, which is the in-process CWD. `-m` stores the actual value of CWD at invocation, so changes to CWD while the program runs don't influence module resolution

Relative imports used to work much more naturally IMHO in python2 but then they broke it in python3 because Guido wanted scripts and modules to always be separate codebases. So, whereas it used to be easy to have a module that could also be run as a script inside a package, this is now very difficult to implement. To the extent that any python2 code that does this, should probably be refactored when being ported to python3.

This decision has bit me in the ass so often.

I want to organize my code logically in directories. As a script grows, I want the ability to spin out parts of that file to separate files.

In order to do that in python, I need separate directories between the script and the spun-out functionality. This ends with a script that says "do function from module" and all code being in the module.

Having code in different directories for no reason except "the import system" sucks. How is this supposed to go?

FWIW, my command-line tools and any support code are all in a subdirectory under my package directory.

I used to include a simple stub Python program (~10 lines) as a "script" in my setup.py that would import the right code and call it.

Then I learned that "entry_points" implemented most of the dispatch behavior I wanted.

I no longer have those scripts, just an entry that basically says "from abc.cli.prog1 import main; main()". The prog1.py, prog2.py, etc. look like normal command-line scripts, assuming the usual:

    if __name__ == "__main__":
        main()

The major downside is I can't suppress SIGPIPE and SIGINT until main() is called, leaving a wider window where something like ^C gives a unwanted Python stack trace.

So as your script grows, don't you eventually rename it to whatever/__main__.py and run it with -m? seems like a fairly trivial transform that then allows you to spin out as many modules as you need.

Are you sure you're not confusing changes to the import system with changes to the default sys.path (especially when executing a file)?

Because unless I'm missing something, the changes to the import system make this more reliable, not less: in Python 2, implicitly relative imports means `import x` might depend on the file's location. In Python 3, it does not, it depends solely on the setup of the sys.path.

Always an informative (and fun) read:

https://stackoverflow.com/questions/14132789/relative-import...

Nice, do you know of anything similarly-comprehensive that is updated for Python 3? IIRC, Python 3 simplified things a good deal by removing "implicit" relative imports, but I'm a little foggy on exactly what that means.

Explicit relative imports use a "." to indicate the file/package is from the current directory and not found elsewhere in sys.path:

  from . import foo
  from .foo import bar

Implicit relative imports don't have such an indicator:

  from foo import bar

In py2, that second one could be a relative import or from anywhere in sys.path, while in py3 those implicit relative imports were removed so that means it'll only only look in sys.path and not the local directory.

https://stackoverflow.com/questions/48716943/what-is-python-...

> IIRC, Python 3 simplified things a good deal by removing "implicit" relative imports, but I'm a little foggy on exactly what that means.

Exactly that: in Python 3, if an import doesn’t start with a . it will always be absolute aka looked up from sys.path.

In python 2, it would try performing a relative import (so importing a sibling module) first.

> the core language seems to wants all packages installed in /usr.

There's also ~/.local/lib/python3/site-packages (or whatever your distribution made of that). Virtualenvs are only necessary if you want to isolate dependencies between environments. That's useful if you have projects with conflicting dependencies, because Python doesn't allow you to install multiple versions of the same package, for better or worse. However, if you've written some simple scripts that don't care much about the exact version of their dependencies, it's perfectly fine to install their dependencies glboally.

Quick shout-out to nix-shell shebangs, which allow a script to specify exact versions of all dependencies, Python or otherwise, which will be cached into a temporary sandbox.

https://nixos.org/manual/nix/stable/#use-as-a-interpreter

I wrote a Python script yesterday which calls out to a couple of external commands (`mlr` and `xq`), with a shebang like this:

    #!/usr/bin/env nix-shell
    #!nix-shell -i python3 -p python3 -p miller -p yq

Earnest question: why are you all trying to use relative imports? What problem is that solving for you? I've never even bothered to try it out because it seems potentially problematic in the way all relative references can be, e.g., relative file paths.

So much this. There's no good reason to use relative imports. They're less readable, more dangerous and don't solve a single problem.

In JS land everything is a relative import. It's nice because you can move a whole directory of code from one project to another (or around in the same project) and it still works because all the imports pointing to other files inside that directory were relative, you only have to fix imports that go outside the directory.

Also the way that JS imports are just relative paths is very nice because it means that the imports are statically determinable, your editor can understand them and fix them automatically and you can trust that refactoring. Python has turing complete imports because there's so much dynamic messing about with sys.path that goes on in Python due to inadequacies of the import system.

Given most editors handle absolute imports as good as if not better than relative imports, I don't see any real benefit to using the latter.

It stops some circular imports if you use relative imports.

I'd rather suffer the late import with absolutes than wrestle with relative imports

> It's nice because you can move a whole directory of code from one project to another (or around in the same project) and it still works

Yes, just like absolute imports...

> relative paths is very nice because it means that the imports are statically determinable, your editor can understand them and fix them automatically and you can trust that refactoring

1) Just like absolute imports?

2) I absolutely contest that relative imports are easier for an IDE to refactor. I've never had VSCode hang while refactoring a Java package; I've had VSCode hang while refactoring "create-react-app" app...

In Python you can do (and people do do) this

    import a.b.c
    sys.path.insert(0, 'some/other/path')
    import x.y.z

    env PYTHONPATH="foo/bar:$PYTHONPATH" python ...

There is no way in general for Python tooling to figure out where an import points. All it can do is guess. It's a very regrettable situation.

Yes in Java it's fine because Java has a build stage. So tooling can figure out where imports point to from your static configuration.

However from the tool's perspective there is nothing better than a relative path. Relative paths require no messing about with configuration files at all to resolve. It's just a path to another file or directory on disk.

When a tool sees

    import c from "./a/b/c"

It can resolve it immediately.

So what is the advantage of absolute imports really? If import lines are mainly written and maintained by tooling then shouldn't we pick the representation that is easiest for the tooling? Then we can have more and better tools and the tools will be more reliable.

And it turns out that, relative paths are easy for humans to understand too. The same configuration-free resolution algorithm also works in your head when you are reading code! At least when the language doesn't overcomplicate them too much (JS is guilty of this to a certain extent, although nowhere as bad as Python)

IIRC, it was done for security, to avoid inadvertently picking up a library from an unexpected place. For instance, drop a file named test.py in the same directory as your python 2 script and have "fun" figuring out what went wrong.

People have a script run with python, and want to use code in other files.

This is not supported in python. For reasons beyond my understanding, you are supposed to put the script with python, or with the shebang, in a different directory.

Alternatively, you can always use `python -m` to run your code.

> People have a script run with python, and want to use code in other files. This is not supported in python.

Come again? This is what "import" does, obviously. This is not a clear explanation of your problem.

I missed one detail. The files and the script are in the same folder.

I say script here because the apparent intention is that scripts and modules are separate. That is why it's not easy to import functions from a file in the same directory as a script.

This restriction is very unexpected. And it is not borne out well by the fact that it is not obvious this distinction exists.

When you run a script, the directory containing it is on the path. You can just write "from somefile import something". What you can't do is "from .somefile import something" because it is not a package.

When you run a module, the root package containing it is on the path by definition and imports work.

I didn't know that.

Does that still work if you call the script from a different directory (so `python bar/foo.py`)?

Imma guess it doesn't work with symlinked scripts. But I don't think it should without doing an install, in which case a module does make sense.

The directory containing the file is added to the path, and the named file is imported as the __main__ module. That's why "if __name__ == '__main__':" works in script files.

I don't think Python will resolve symlinks here, so if that needs to work, you probably need to do it yourself before touching other code / files.

While I rarely need it, I like being able to vendor a package by dropping it into a project subdirectory.

If the package uses absolute paths then I would need to tweak all the imports for the new absolute path.

I do use them quite often without much issue, not sure why some people are struggling with it.

Say you have a main package "mylib" with a subpackage "mylib.utils". Typically I like to see "mylib.utils" imports as being in one of (roughly) 4 categories:

- standard imports, that I would put first in the file (e.g. "import logging")

- external imports, that I would put second in the file (e.g. "import requests")

- library local imports, bits and pieces from the "mylib" package that you want to reuse in "mylib.utils", but are external to the current package (e.g. "from mylib.email import client")

- and package local imports, which I see as implementation details of the current subpackage, and should be agnostic from the overall architecture of "mylib" (e.g. "from .helpers import help_function")

The last category are modules that only make sense from within "mylib.utils", should relocate with it even if it is renamed or moved elsewhere, and shouldn't require a change whatever the structure of "mylib" becomes, which is why I would use "mylib.utils" relative imports in there.

> Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH.

Well whatever the language or runtime, you need to tell it how to find its dependencies. Be it with a configuration file, environment variables or parameters.

Your statement is not exactly correct either. You don't need a fake installation _and_ bend PYTHONPATH.

Virtualenv leverages the fact that "../lib/python<version>/site-packages/" relative to the interpreter is a default import path (default value of PYTHONHOME). It doesn't use PYTHONPATH.

> Why couldn't the import system resolve versions, too?

Not sure that would be really great. I prefer to have my dependencies all grouped together in a setup.py rather than scattered in various files at import time.

> Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH.

If I understand what you mean by package management in this context, I wonder if editable installs will help you.

https://www.python.org/dev/peps/pep-0660/#abstract

https://discuss.python.org/t/pronouncement-on-peps-660-and-6...

> Why couldn't the import system resolve versions, too? You could say `import foo >= 1.0` and it would download it to some global cache, and the import the correct versions from the cache.

What do you do about conflicts? Or say you have `import foo >= 1.0` in a file, and `import foo == 2.4`, but the latest version is 2.5, so the first import grabbed the latest version, and you later realize you need 2.4?

Imagine running a report generator for 5 hours, only to have the formatting module require a conflicting version of something and erroring out at run time...

> Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...).

I think the thing to remember about using relative imports is that it requires modules. Using relative imports to import something into a script will fail because the scripts don't belong to modules.

> Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. Virtualenvs are the canonical way to do it, but to me it feels like a hack - the core language seems to wants all packages installed in /usr. So now I have all these virtualenvs lying around and they are detached from the scripts.

This is a pain. You can abstract this away with with pyenv or virtualenvwrapper, though.

I never followed the changes around imports in 3 … but def still run into issues especially trying to move code from dev to deploy

Yeah, saner relative imports might be one of the few things that I envy the JS ecosystem for.

Reply on news.ycombinator.com