Don't get me wrong, I LOVE Python! It's my go-to language for both small and big tasks. But sometimes I do wish Python had a function like PHP's require()… (Obviously for package-local imports, not for importing other packages.)
Don't get me wrong, I LOVE Python! It's my go-to language for both small and big tasks. But sometimes I do wish Python had a function like PHP's require()… (Obviously for package-local imports, not for importing other packages.)
project
├── main.py
├── package1
│ ├── module1.py
│ └── module2.py
└── package2
├── __init__.py
├── module3.py
├── module4.py
└── subpackage1
└── module5.py
As long as the file containing the main() function resides in the top-level folder you can use absolute imports in every file, e.g. from package1 import module1
from package1.module2 import function1
from package2.subpackage1.module5 import function2The only way this can be accomplished is by designing the import system from the start to use a function instead of a statement. It's too late for Python to redo things and change its import system, unfortunately. It really is the one language feature you have to get right the first time or you'll miss out on those kinds of enhancements forever.
[0] https://en.m.wikipedia.org/wiki/Live_coding#Notable_live_cod...
Why not just replace the `__import__` function at program startup, when you haven't imported anything? Monkey patching `__import__` at any other time seems like a bad idea.
The docs for Python's importlib say as much:
> If a module imports objects from another module using from … import …, calling reload() for the other module does not redefine the objects imported from it — one way around this is to re-execute the from statement, another is to use import and qualified names (module.name) instead.
But Lua isn't actually different here since you can still bind the members of an imported module to local names manually, it's just that there is no "from" keyword that does this for you. My personal theory is because of the lack of something like "import * from" in Lua, it became more common for library authors to use late binding and access the returned module by indexing into it. However this is just my personal theory.
Also, now that I think about it you still have to make adjustments to your code in order to have hotloading be feasible, even in Lua. I deliberately chose to use late binding in the code I wrote with "qualified names" (really just table indexing). And all the third party dependencies are versioned in the project's repository, and don't have dependencies on other external modules, so there was no need to worry about a dependency breaking hot reloading because it uses imports differently. I think this might be a cultural thing. A lot of Lua libraries are written in a single-file, no-dependency manner which makes them much easier to integrate in the ways I want. Python seems to have a much larger ecosystem with many external dependencies depending on others.
So yes, I think it makes sense that Python could have module reloading, but the problem is that the language encourages the use of local binding from imports which is incompatible with it, and there is already a lot of code that imports things like that, so it's not as practical if you're using libraries from pip or use "import <...> from" anywhere in your codebase.
Edit: the below is probably wrong about what you're doing. It sounds like your require-replacement merges the new return value of a module into the old one, so code that accesses stuff through the module's top level table will get the right result.
Wrong stuff: Maybe the hot reloading system you use relies on modules mutating the module object rather than making a new one. This is pretty unusual among Lua modules though. I usually see people unconditionally make a table and return it.
from importlib import reloadhttps://docs.python.org/3/library/importlib.html#module-impo...
So in many instances, there's no need to replace the require builtin: one may simply add a new function to `package.loader`, anywhere in that order that's useful.
I've done this, and draw most (basically all) modules out of a SQLite database. It works like a charm.
If one has need to invalidate the cache for a given module, as comes up in live coding environments, simply set the require string in `package.loaded` to `nil`. Small caveat: the `package.loaded` table is 'special', in that the C which `require` is written in hard-codes a reference to it: one may not simply replace `package.loaded` in the global namespace and expect the new one to be referenced.
Python’s import goes through the configured finders (sys.meta_path) and invokes them in-order with the name of the package being imported until one of them returns an import spec. Sound ‘bout the same to me?
I’ve written loaders which created “virtual” modules on-demand in the past.
I regularly have to spend some time on pandas source code just to untangle the mess of import aliases magic they do.
But the standard library is no exception, importing os will dynamically import os.path.
Now that I do everything in a container, life is much improved.
Try Docker as documented in the AllenNLP README:
$ mkdir -p $HOME/.allennlp/
$ docker run --rm -v $HOME/.allennlp:/root/.allennlp allennlp/allennlp:latestIf I started a new job or new project, it was the same story every time. “Do these five things and then you should be able to run the code” after half a day of back and forth I’d have a list of ten things, and the next new person would discover it is actually 12.
Docker doesn’t just change your experience, it forces the author to actually capture most of their prerequisites. To the point where even if you don’t use docker, you benefit from a project having at least tried to use it.
https://github.com/tensorflow/tensorflow/blob/master/tensorf...
Inside the source code files, they then use a decorator (@tf_export) to define where in the package/module hierarchy a function/class should get placed, see e.g.
https://github.com/tensorflow/tensorflow/blob/master/tensorf...
For instance, the write() function in that module actually gets exposed as `tf.summary.write()` even though it's contained in the file tensorflow/python/ops/summary_ops_v2.py.
Needless to say, this even brings auto-completion of an IDE like IntelliJ to its knees at times.
To make things even worse, things like Tensorboard also make use of global state everywhere, see e.g. the function _should_record_summaries_internal() in the above file. But now the question is: Which module's global state does the function actually access if it gets "moved" to a different module by TF? At the end of the day, I gave up hunting down that bug (_should_record_summaries_internal() always returned False, no matter what) and resorted to mocking the function out whenever I used Tensorboard. To do that (i.e. to find its parent module in the first place), I in turn used the `inspect` package.
This is what I meant by "import magic". :)
I’ve never had a problem with python’s import system either.
I ask because from what I understand from Google's monorepo, it should be easy for engineers to take direct dependencies on whatever they want.
Of course TF is an exported out library so things might be different, but surely it won't then be a good example of Google's DNA?
Not correct. The dependency system supports visibility restrictions - you can take direct dependencies to anything only if you're in your own experimental tree. Otherwise package authors can control access.
You can see that even in the Bazel: https://docs.bazel.build/versions/master/visibility.html
(google employee)
It's not like most people would want more complexity (at least I hope not), but somehow it kept happening, and I feel at least some of it is cultural.
1) They hire a lot of smart but inexperienced people. I've noticed that those kinds of programmers often write clever and overengineered things, which means overly complex.
2) They get promoted for creating complicated things. They don't get promoted for writing simple things.