How the Python import system works

It starts with reader.l: https://github.com/shawwn/pymen/blob/ml/reader.l where the raw character stream is turned into a bunch of nested arrays. E.g. (+ 1 2) becomes ["+", 1, 2]

Then it's punted over to compiler.l https://github.com/shawwn/pymen/blob/ml/compiler.l where it's passed through `expand`, which does a `macroexpand` followed by a `lower`. E.g. (do (do (do (print 'hi)))) becomes ["print", "\"hi\""]

Then the final thing is thrown to the compile function, which spits out print("hi") -- the final valid Python code that gets passed into the standard python `exec` function.

Works with all the standard python things, like async functions and `with` contexts. Been screwing around with it for a few years.

Should work with anything as long as you can ultimately generate Python bytecode (and provide a module object). The import system is not simple, but it's really open.

But, I did make an interactive tutorial here: https://docs.ycombinator.lol/

If you have any questions about it, I'd be happy to answer. This stuff is pure fun mixed with a shot of professionalism.

For what it's worth, as someone with narcolepsy, I relate quite a lot to your chronic pain. (https://twitter.com/theshawwn/status/1392213804684038150) For me, it mostly translated into wandering aimlessly from job to job, since I thought no one would have me. I hope that you find your way -- there's nothing wrong at all with taking it slow and spending years on something that takes others a few months. Everyone is different, and it's all about the fun.

  git clone https://github.com/shawwn/pymen -b ml
  cd pymen
  touch foo.l
  bin/pymen
  (import foo)

You need nodejs to be installed too, ha.

I mean, it is used—even in the standard library—but often for alternative packaging (e.g., loading python modules in zip files) rather than alternative languages. It may be used less prominently than in Node, but it definitely is used for a variety of things.

https://nixos.org/manual/nix/stable/#use-as-a-interpreter

I wrote a Python script yesterday which calls out to a couple of external commands (`mlr` and `xq`), with a shebang like this:

    #!/usr/bin/env nix-shell
    #!nix-shell -i python3 -p python3 -p miller -p yq

This.

IMO that's also one of the main issues of Bash – you can't modularize a script unless you make sure your working directory is the directory containing the script. (And good luck with finding out the latter! [0])

[0]: https://stackoverflow.com/questions/4774054/reliable-way-for...

https://pastebin.com/5WCXb2Gg

Assuming `foo.bar` is a script inside the package `foo` that you want to both import and run directly (without `python -m ...`), this lets you do so without too much hassle.

It does not though, unless you have altered the default sys.path to always contains `.`.

When running a file / script, Python will add that file's directory as the first lookup path, so these two invocations should have the exact same sys.path, and thus the same module resolution throughout[0].

If you have added `.` to sys.path (which is not the default), then the first invocation will also have the "grand-parent" folder on the sys.path, while the second won't.

This also doesn't seem to have anything to do with relative imports, you need to already be inside a package for relative imports to do anything.

[0] that is not the case of `-c` and `-m`, both add CWD to the path, and they differ in what they do: `-c` adds `''` to sys.path, which is the in-process CWD. `-m` stores the actual value of CWD at invocation, so changes to CWD while the program runs don't influence module resolution

I want to organize my code logically in directories. As a script grows, I want the ability to spin out parts of that file to separate files.

In order to do that in python, I need separate directories between the script and the spun-out functionality. This ends with a script that says "do function from module" and all code being in the module.

Having code in different directories for no reason except "the import system" sucks. How is this supposed to go?

Because unless I'm missing something, the changes to the import system make this more reliable, not less: in Python 2, implicitly relative imports means `import x` might depend on the file's location. In Python 3, it does not, it depends solely on the setup of the sys.path.

This is not supported in python. For reasons beyond my understanding, you are supposed to put the script with python, or with the shebang, in a different directory.

Alternatively, you can always use `python -m` to run your code.

If the package uses absolute paths then I would need to tweak all the imports for the new absolute path.

Say you have a main package "mylib" with a subpackage "mylib.utils". Typically I like to see "mylib.utils" imports as being in one of (roughly) 4 categories:

- standard imports, that I would put first in the file (e.g. "import logging")

- external imports, that I would put second in the file (e.g. "import requests")

- library local imports, bits and pieces from the "mylib" package that you want to reuse in "mylib.utils", but are external to the current package (e.g. "from mylib.email import client")

- and package local imports, which I see as implementation details of the current subpackage, and should be agnostic from the overall architecture of "mylib" (e.g. "from .helpers import help_function")

The last category are modules that only make sense from within "mylib.utils", should relocate with it even if it is renamed or moved elsewhere, and shouldn't require a change whatever the structure of "mylib" becomes, which is why I would use "mylib.utils" relative imports in there.

The common approach to solving this is pulling everything that is used by all the modules into leaf libraries, effectively creating a directed acyclic graph, but this is not obvious nor easy to do the first time.

If you keep doing that for a while, a circular dependency will happen.

This is the dumbest thing thing in Python. All other languages I know have solved it.

What is the purpose of modules? You may as well collapse into a single script and the situation would not be any different, would it?

I'm not being facetious here. The answer to the second is the answer to the first.

A common example might go like this. You have a module for each kind of thing you have in the database. But now if someone loads a Company, they need to get to Employees. And if someone loads Employee they need to get to Accounting for the salary, payments, etc. And Accounting needs to be able to get to Company.

Those are all large and complicated enough that it makes sense to make them modules. But you just created a circular dependency!

The standard solution is to load a base library that loads all kinds of objects so they all can assume that all the others already exist and don't need a circular dependency. But of course someone won't like wasting all that memory for things you don't need and ...

Suppose you have some method of A which does something special if it gets an instance of B, and vice versa. Now you have a circular import problem; glhf

Boom, circular dependency.

Happens in basically all corporate code bases that grow over the years, with varying path lengths.

Throwing all potentially circular types into one big module isn't a great solution.

(In practice, we tend to rely on run-time imports to make it work. Not really great, but better than throwing several 10k or 100k lines of code into a single module).

Circular imports are only ever a problem when you have code running when the module loads. Then you run into module load ordering issues. So avoid any side effects on module load and make all setup explicit.

For adding current directory to front of path, the sys.path.insert() call is a pretty sound way of doing it.

    #!/usr/bin/python

    #!/usr/bin/env python

The contraption I wrote allows adding arbitrary parameters - I was burnt one too many times by Python silently buffering my debug messages, so I use it to always add "-u".

For additional bonus points, have your single entry point exhibit a CLI that properly documents everything the developer can do, i.e. what features are available, what environment variables and config flags can be set etc. That way, the code essentially documents itself and you know longer have to keep your README file updated (which people always tend to forget).

- everything is structured as a module

- options and args are stored in their own module for easy reuse

- the whole stack has one cleo.Application, with however many subcommands. Usually of the form "mytool domain verb" e.g. "mytool backend start."

- cleo args/options are parsed into pydantic objects for automatic validation (you could do this with argparse and dataclasses to skip the deps but it's more work)

- each subcommand has a `main(args: CliArgsModel)` which takes that parsed structure and does its thing. This makes it super easy to unit test

I install into a venv with `poetry install` or `pip install -e` for editable installs.

It all just works, no fuss, and so damn modular.

I'm sure if I'd taken the time to try and fix it I eventually could have and at this point i've had more experience with a bunch of different languages, so I'm sure it's not as bad as I remember.

I imagine it's one of those cases where if i were to go back and laugh about how stupid I was, but ya know, those first impressions.

Skip the relative imports in the article. No need to read it entirely anymore ;-)

I now have the rule of, only ever use pip inside a venv. If your venv is more than a little bit complex, write a requirements.txt file so you can generate it. So it's something like

    $ cat > requirements.txt << EOF
    tensorflow
    Django==3.2.5
    cryptography
    EOF
    $ echo venv > .gitignore
    $ python3 -m venv venv
    $ venv/bin/pip install -r requirements.txt
    $ venv/bin/python

    $ . venv/bin/activate
    (venv)$ pip install -r requirements.txt
    (venv)$ python

    $ rm -r venv
    $ python3 -m venv venv
    $ venv/bin/pip install -r requirements.txt

Either of these rules works fine. The thing that works poorly is using pip install outside of a venv (with or without root).

It's nice to have something that "just works" without having to install it. I like to call it Level 5 autonomous software -- software that can figure out how to run itself with zero complaints.

I actually use this for a lot of personal non-production scripts, I can just clone the script on any system and just run it, it will figure itself out.

Also, packages with version dependencies fuck up other packages with other version dependencies, unless you set up virtualenvs or dockers or condas for them, and those take additional steps.

magicimport.py uses virtualenv behind the scenes, so when it imports tornado 4.5 because some script wants 4.5, it won't mess up the 6.0 that's already on your system. In some sense it just automates the virtualenv and pip install process on-the-fly, to give the effect that the script "just works".

Start with pip, the official package manager. You don't have to start with virtual envs: I didn't.

I haven't ran into any function which I "should not use".

You don't need an entry point, you can just write code in a file. Besides that, I don't see how much it differs from other languages with implicit entry points, where you need to match a certain name in your function.

Imports: yes, those are annoying and confusing. I still struggle with those.

Aditionally: package managers and virtual envs are a pain in the ass. Every year we're getting a new one which is supposed to solve the problems from the previous one, but doesn't, and the cycle goes on. The language should really solve this at the core instead of requiring community fixing, as it is a core part of any serious development.

edit: Actually I'm not even sure what kind of error we're talking about here. If two modules import each other and they both need access to the other's contents upon initialization, there is no ordering that will work. And if at most one needs access to the other, it will always work, no matter in which order they are imported. So I don't really know what the OP was talking about.