Traps for the Unwary in Python’s Import System

I left one company 3 years ago in the middle of a major transition between versions. The "right way" changed fairly drastically, and surprise surprise, they're still stuck in the middle of that transition.

I think this is largely why I have found myself preferring static typing with roll-your-own framework style / library-based ecosystems over batteries-included systems.

When your language or framework of choice suddenly insists that you use AA batteries instead of button cells, and it is up to you to modify every electronic device in your house to use them, it gets real old real fast (an exaggerated example, but not entirely unheard of).

Really should be, write stuff and test it locally on my machine, put it in a container and upload it. I know Lambda supports containers but sometimes I feel like its too much of an overhead, especially with shitty upload speeds but I really don't see any other choice unless I manually build AWS Lambda layers myself.

In 2022, If you're building an application and are looking for a simple answer, just use Poetry; it's similar enough to what you'll be familiar with if coming from other languages (e.g. yarn, cargo, ...). For most usecases you can just lean on Poetry's venv management; you don't need to do more than `poetry shell` or `poetry run` to get access to it. Poetry will create, and for the most part manage, your pyproject.toml file for you.

If you want a bit more flexibility Virtualenv is part of python now (`python -m venv`). If you're on MacOS you can get great venv management with pyenv and pyenv-virtualenv. And for sure, things get more complex if you're building a library. But I think Poetry is a solid place to start.

  import hello.world.bye as bye

Edit: While opinionated, it's a real question; in my fairly simple use of Python, I have never hit a use of "from" that couldn't have been a plain "import", but if I'm missing a case feel free to correct me.

I find that's kind of the point. Python was extremely easy the first time,

Learning Ruby took me twice as long and the output was not really an improvement.

Python looks for regular packages (which are really just modules that can contain other modules) by searching a pre-defined list of directories, stopping at the first valid entry.

What happens when you run `import foobar`?

Let's say that the search path is:

    sys.path = [
        # The current working directory
        '',
        # System package locations
        '/usr/lib/python3.9/site-packages',
        '/usr/local/lib/python3.9/site-packages',
    ]

    /usr/lib/python3.9/
    └── foobar/
        ├── __init__.py
        └── nice.py
    
    /usr/local/lib/python3.9/
    └── foobar/
        ├── __init__.py
        └── wow.py

Now let's assume that you didn't have the `__init__.py` files:

    /usr/lib/python3.9/site-packages/
    └── foobar/
        └── nice.py
    
    /usr/local/lib/python3.9/site-packages/
    └── foobar/
        └── wow.py

The use case for this feature is the ability to split a package over several distributions. For example, you can install one or both of zope.interface and zope.component, and they will both be available to import from the "zope." namespace. It is also useful within organizations, where you can namespace all of your internally-developed libraries with some identifier specific to your organization.

You could even emulate a reverse-domain namespacing system this way if you wanted to:

    /usr/local/lib/python3.9/site-packages/
    └── org/
        └── best_stuff/
            └── pytools/
                ├── thing1.py
                └── thing2.py

As you might imagine, spreading a package across multiple locations is not a desirable outcome if you aren't expecting it. So you shouldn't use namespace packages unless you specifically want the behavior I described above.