Hacker News Reader

Comment by rad120 | Hacker News Reader

[flagged]

> Python is such a weird language. Lazy imports are a bandaid for AI code base monstrosities with 1000 imports

Just because you don’t like a feature doesn’t mean it’s because of AI and bad code.

I think this is just a natural consequence of an easy-to-use package system. The exact same story as with node. If you don't want lots of imports, don't make it so damn easy to pile them into projects. I'm frankly surprised we still see so few supply chain attacks, even though they picked up their cadence dramatically.

This seems a lot more due to an import running arbitrary code because stuff can happen in the top-level of a module rather than only happening in functions. From what I can tell, it seems pretty common for dynamically typed languages and pretty much entirely absent from statically typed ones, which tend to have a main function that everything else happens inside transitively. I guess this makes it easy if what you're writing is something that runs with no dependencies, but it's a pretty terrible experience as soon as you try to introduce the concept of a library.

> it seems pretty common for dynamically typed languages and pretty much entirely absent from statically typed ones

Counter-example is Go and init() function.

Static initializers in C++ - sometime ago I saw savings of some 400 ms (?) startup cost of initializing static strings from constants by moving it to some compile time thing.

Right; the issue is that this isn't happening at compile time in Python, because it's not getting compiled ahead-of-time. The equivalent would be if header files had imperative code that got executed at runtime in places where they're included.

(To preempt potential pedantry: yes, I know that you can compile Python to bytecode ahead of time, but that's not really relevant to what's being discussed here because it doesn't mean "the stuff happening in modules I import isn't happening at runtime anymore")

> the issue is that this isn't happening at compile time in Python, because it's not getting compiled ahead-of-time

What are those .pyc files for?

That was the pedantry that they were trying to preempt.

Interesting, I had no idea that existed! I still think there's a a difference between "here's a hook you can use to run stuff earlier" and "importing any module is fundamentally the same as running it as a script unless the module happens to use a special conditional to wrap stuff inside of" though (and I say this as someone who doesn't go out of his way to defend Go design decisions)

Also C++/Java static initialization, C# static constructors, or Rust global variable initialization, ...

Most languages have this feature Afaik

Rust doesn't have this behavior (sometimes called "life before main"). Code to initialize a static variable runs either at compile time, or lazily on first access, depending on which mechanism you use.

Interesting, thanks

Yeah, I don't think that "precompute something at compile-time" is really comparable to "every import literally executes code as a script". Rust imports are actually about as far from this as I can imagine, because modules can circularly reference each other, which unless I'm misunderstanding would be an infinite loop in Python without manually breaking the chain with some form of conditional.

I'm actually kind of surprised to see comments like that one, because compile-time logic feels like the opposite end of the spectrum from what Python imports do, with "regular" code being compiled without any precomputation sitting somewhere in the middle. It seems like I didn't articulate my thoughts clearly enough though, since several people seemed to read what I was saying as being comparable.

IIUC the organizations that most strongly pushed for this feature are big companies with large codebases. These tend not to be the kinds of orgs that just casually pull in dependencies from PyPI on a whim; I think it more likely that the quantity of first-party code was so large that importing all of it on startup was causing problems.

I worked in a codebase like this. The load time was insane.

We would also constantly need to put imports in function heads to effectively lazy import due to the massive risk of circular imports. We also had dynamic imports so tracking the cycles was very difficult at times.

What would your alternative look like?

Too much syntactic sugar causes cancer of the semicolon.

True, but this is yet another code path that isn't exercised until specific conditions happen. That means even more latent application behaviour can go undetected by unit testing and security profiling until the moon is in the right phase, which is a boon for submarine attacks.

Empirically, I have used the current accepted way to do lazy imports (import statement inside a function) before AI coding was even a mainstream thing, for personal code that sometimes needs a heavy import and sometimes doesn’t.

The lazy statement would be an improvement as it allows one to see all the imports at the top where you expect them to be.

As a now deleted comment pointed out, lazy imports had been requested forever. They were rejected forever and were accepted just when BigCorps wanted them.

Python-dev now is paid to shore up the failed Instagram stack.

both lazy imports and free threading have been proposed ages ago, they both went through several iterations before a good design was settled upon and made it into the language.

in the case of lazy imports the big corps were the ones doing the experimentation and iteration. the feature didn't make it into the language "just when big corps wanted them"; the instagram stack you allude to already had its own fork of cpython with lazy imports added years ago, and that is not the design that ended up getting adopted by upstream cpython, though some of the people working on it also collaborated on the PEP that finally did make it in.

It was accepted just as multiple large corporations with competent teams of internal tool departments ended up forking the interpreter to support lazy imports and demonstrated empirically that the idea has merit.

I too am outraged that a product would prioritize its biggest users.

Is the biggest user larger than the combined set of individual users who had asked for (or would benefit from) the same thing? I honestly don't know, but I don't think that things are always as simple as you're implying in a world where we have the collective action problem.

If you’re asking some some kind of abstract moral value sense, I have idea.

If you’re asking whether project leads give more weight to a single, tangible, vocal stakeholder than they do to unknown numbers of anonymous and lightly-engaged stakeholders? Yes.

Not to mention when the single, tangible, vocal stakeholder can also be asked to be responsible for documentation (PEPs, etc) and PRs. Especially in open source there is a huge difference between "a lot of people asked about this" and "one person asked about this, but was passionate enough about it and open enough to following the process and the feedback loops to champion it all the way across the finish line".

I don't have any issue with what you're saying if that's what happened. There's quite a gap between that sort of reasoned explanation and treating concerns about large stakeholders versus large numbers of small one with derision.

For what it is worth, I was trying not to make a value judgment on it, especially not with relation to this specific instance, I was hopefully just recognizing it as a motivating factor in general open source politics. Sometimes that is quite regretful because it is anti-democratic and does look like favoritism or worse cronyism when it plays out in that way of "we listened to the person/company that built and tested a prototype and did all the work to standardize and then PR it over the many developers that wanted an idea but didn't have the time/money/bandwidth to implement it themselves".

That's fair. I think I mostly reacted because of the sarcastic faux-outrage that the original comment I responded to expressed. These are hard problems, and I think the presumption should be someone being frustrated at the slow state of changes they want probably has legitimate reasons to feel that way, just as the presumption should be that open-source projects that have run successfully for a long time probably are making good-faith effort to steward what they're maintain. Acknowledging the tension between priorities not lining up exactly for everyone and not having knee-jerk reactions when someone is unhappy seems preferable to mocking those who you disagree with.

I mean, yes, demonstrably, the phenomenon you're describing happens. Your previous comment seems pretty sarcastically dismissing the idea that someone could disagree with this being a good thing though, and I was making a counterargument against the underlying opinion that seemed apparent.

On most unix-likes all "imports" via shared libraries (e.g. in C / C++) are lazy by default.

Eh, resolving object symbols is something done at runtime. #include absolutely is eager.

I wouldn’t compare this in any way to Python’s lazy imports.

Lazy in the sense that by default calls to external symbols will jump through the PLT which will jump into ld and resolve the symbol during the first call to the symbol (ld then patches the PLT to point to the actual function for later calls). Not lazy in the sense that shared objects are resolved at runtime, which is existential to dynamic linkage.

Deleted comment

But also great for speed. Larger libraries can take a measurable amount of time to import (even if they have no transitive dependencies). If only some of your code paths actually need the large library then it makes sense to import it lazily. Without lazy you have to do it conditionally which can lead to the imports happening in strange places rather than all being listed out at the top of the file.

Reply on news.ycombinator.com