Hacker News Reader

Comment by simonw | Hacker News Reader

Love this. My https://llm.datasette.io/ CLI tool supports plugins, and people were complaining about really slow start times even for commands like "llm --help" - it turned out there were popular plugins that did things like import pytorch at the base level, so the entire startup was blocked on heavy imports.

I ended up adding a note to the plugin author docs suggesting lazy loading inside of functions - https://llm.datasette.io/en/stable/plugins/advanced-model-pl... - but having a core Python language feature for this would be really nice.

View on HN

You can implement this from your tool today: https://news.ycombinator.com/item?id=45467489

Note that this is global to the entire process, so for example if you make an import of Numpy lazy this way, then so are the imports of all the sub-modules. Meaning that large parts of Numpy might not be imported at all if they aren't needed, but pauses for importing individual modules might be distributed unpredictably across the runtime.

Edit: from further experimentation, it appears that if the source does something like `import foo.bar.baz` then `foo` and `foo.bar` will still be eagerly loaded, and only `foo.bar.baz` itself is deferred. This might be part of what the PEP meant by "mostly". But it might also be possible to improve my implementation to fix that.

Note the PEP does have a FAQ entry that mentions reasons they believe this proposed solution might be preferable to LazyLoader

https://pep-previews--4622.org.readthedocs.build/pep-0810/#f...

Q: Why not use importlib.util.LazyLoader instead?

A: LazyLoader has significant limitations:

Requires verbose setup code for each lazy import.

Has ongoing performance overhead on every attribute access.

Doesn’t work well with from ... import statements.

Less clear and standard than dedicated syntax.

Thanks for pointing it out!

> Has ongoing performance overhead on every attribute access.

I would have expected so, but in my testing it seems like the lazy load does some kind of magic to replace the proxy with the real thing. I haven't properly dug into it, though. It appears this point is removed in the live version (https://peps.python.org/pep-0810).

> Doesn’t work well with from ... import statements.

Hmm. The PEP doesn't seem to explain how reification works in this case. Per the above it's a solved problem for modules; I guess for the from-imports it could be made to work essentially the same way. Presumably this involves the proxy holding a reference to the namespace where the import occurred. That probably has a lot to do with restricting the syntax to top level. (Which is the opposite of how we've seen soft keywords used before!)

> Requires verbose setup code for each lazy import.

> Less clear and standard than dedicated syntax.

If you want to use it in a fine-grained way, then sure.

Is it another potential solution (until PEP 810 is accepted) to override the NameError exception, decide if it was triggered by an unloaded package from a list, and then running again that line of code? I understand the inefficiency of this solution (e.g. the same line could trigger NameError several times and you need to run it again until all modules are loaded) but this is a good brainstorming thread.

That sounds very unpleasant. However nicely you wrapped it up, you'd still be referring to the code for that process everywhere that the NameError could occur.

Parse the command line and do things like "--help" without doing the imports.

Only do imports when you know you need them -- or as an easy approximation, only if the easy command line options have been handled and there's still something to do.

Well yes, or you can just use the `lazy` keyword, when it makes it into core.

You can “just” use a feature which does not exist yet? How is that something you “just” do?

  lazy from __future__ import __lazy_import__

I think Guido's time machine will need some serious overclocking to handle that one!

In the llm project, plugins can modify the command line arguments, so it's not that simple.

Yea, that's the core problem here: plugins can add new CLI subcommands, which means they all need to be loaded on startup.

https://llm.datasette.io/en/stable/plugins/plugin-hooks.html...

Could you cache the help doc after first full loaded run and only regenerate when new plugins are added / updated?

It's not just help - the plugins need to be imported so the root level CLI tool knows what to do if you type "llm subcommand ..." where that subcommand is defined by a plugin.

In that case the CLI only needs to import the plugin that defined that sub command, not all plugins?

It doesn't know which plugin defines a subcommand until it imports the plugin's module.

I'm happy with the solution I have now, which is to encourage plugin authors not to import PyTorch or other heavy dependencies at the root level of their plugin code.

> It doesn't know which plugin defines a subcommand until it imports the plugin's module.

That might be considered a design mistake -- one that should be easy to migrate away from.

You won't need to do anything, of course, if the lazy import becomes available on common Python installs some day in the future. That might take years, though.

Just FWIW, a trick that I'm planning to use for PAPER: first I make separate actual commands — `paper-foo`, `paper-bar` etc. that are each implemented as separate top-level scripts that can import only what they need. Later, the implementation of `paper foo` has the main `paper` script dynamically look up `paper-foo`. (Even a `subprocess.call` would work there but I'd like to avoid that overhead)

Or require plugins to be competently written.

Bad performing third party plugins are user error.

I think really the problem is that packages like pytorch take so long to import. In my work I've tried a few packages (not AI stuff) that do a lot of work on import. It's actually quite detrimental because I have to setup environment variables to pass things that should be arguments of a setup function in. All things considered a python module shouldn't take any noticeable time to import

It's not only in the case of plugins.

If a tool has different capabilities that use different imports, why load all of them if only a subset is required?

As a simple example, a tool that can generate output in various formats (e.g., json, csv, xml, ...) should only import the appropriate modules to handle the output format after having determined which ones will be used in this invocations.

Reply on news.ycombinator.com