The Trouble with __all__
gauge.sh
gauge.sh
And please, just don't use * imports. It really doesn't save you much time at the cost of implicit untraceable behavior. If you don't worry about * imports, you don't need to add the __all__ boilerplate to every module.
This article is more about advertising a package called tach, that I suppose tries to add "true" private classes to Python.
But it doesn't actually enforce anything, because you still need to run their tool that checks anything. You could just easily configure your standard linter to enforce this type of thing rather than use a super specialized tool.
If module A has a small intended public API, you can structure it no matter how you want to achieve that. You can put those internal symbols behind their own object/class/module if you prefer.
Using `__all__` has one functional consequence, which is `from A import *`. Again, I would avoid * imports entirely, but if you want to try to curb possible downstream problems from users who do indeed use * imports, I would also prefer not defining `__all__` because it's extra boilerplate you have to maintain and can very easily be missed on future updates.
Relative path imports are unnecessarily difficult, especially if you want to import something from a parent directory. There's no explicit way to define what you'd like to export either.
The syntax is inconsistent, too:
from X import Y
import Z
import { Y } from 'X';
import * as Z from 'Z';
Python imports made sense straight away - but then I've been a Python programmer for a long time so I'm far removed from a beginner's perspective.
Well, not a major headache. I can always revert to the py2 way: symlinks.
Instead of creating "myproject.py", create a myproject folder with a __main__.py file. Run it as "python -m myproject".
Hardly any scaffolding and your code is in a package. It's one of the things I like about python, it's quite easy to move "up the ladder" scaffolding-wise if you need to.
import * as bindingName from 'module-url'
console.log(bindingName.exportedName)
Maybe you meant bare imports?
import 'module-url'
In Python, the following line pairs are roughly equivalent:
import X
X = __import__("X")
import X as Y
Y = __import__("X")
const std = @import("std");
const parseInt = std.fmt.parseInt;
Circling back to Python, the only practical place I see 'import * from ...` being used is in libraries where they are bundling up names from a bunch of submodules to be accessible at a top-level. Though, I was searching for an example of this as I know NumPy did it until they undid it: https://github.com/numpy/numpy/pull/24357
That is exactly how CommonJS worked for the longest time:
const module = require('url')
module.exports.value = 'foo'
import users.*
import users.User
import users.{User, UserPreferences}
import users.UserPreferences as UPrefs
This is never explained well to beginners but this is because Python imports deal with modules and packages, any relationship with directories is accidental. With namespace packages foo.bar and foo.baz might live on totally separate parts of the filesystem. They might not live on the filesystem at all and instead be in a zip file.
somedir/
otherdir/
__init__.py
foo.py
scripts/
__init__.py
myscript.py
# in myscript.py
from .. import otherdir.foo
When you run python -m scripts.myscript it sets __package__ to scripts.myscript so now it does have a parent module and somedir and otherdir are importable under the root module because '.' is added to the search path by default.
I've been told it's because of some deep part of Python interpretation that just can't be changed. The mutable default argument madness is also caused by that.
Anyone know more?
Python does a lot when it evaluates an import statement. That is where a lot of the python magic happens. As soon as you try to limit the import statement somehow, a lot of python code needs to be re-written (and maybe doesn't work at all anymore). That's arguably a bad decision but it's one at the center of the language and it's unlikely to change.
Not quite - it also checks whether the file has already been imported, and does nothing if so.
And it also checks if the file has been "partially imported" which is what causes it to fail during circular imports.
You could imagine a small change to Python which made the second case a no-op as well. This would allow you to use circular imports. You'd need to restrict yourself to imports like "import mymodule; mymodule.myfunction()" rather than "from mymodule import myfunction; myfunction()" but that's encouraged in popular style guides [0] anyway. By the time you run the function, mymodule.myfunction will be bound correctly and everything will work.
This would create hard-to-debug situations for people who rely on executing lots of side-effect code to set up state at import time, but 1) that would only happen in code that currently can't run at all, and 2) those people deserve it.
I think it would even be reasonable to make the "from mymodule" version work by binding those names later, but it wouldn't be a one-line change to the interpreter and it might break some legitimate use cases where a name got redefined.
def foo(bar=[]):
bar.append('lol')
return bar
But if it was viewed as a major language bug, I think they'd find a way.
So it's not a bug, it's one of the central choices that makes python "pythonic."
Every Python project contains a hidden and deadly complexity that will grow over time - and will eventually destroy it. There's no way around it, it creeps in no matter what you do. The imports situation is only part of it - it wasn't what killed our simulation tools, or our build scripts, or our test framework - and required that we rewrite them all in different and more suitable languages.
Python's performance, global modules, whitespace, untyped-by-default code are all killers. You pretty much have to use virtual environments to permit isolation between the multiple differently-versioned sets of dependent packages that you'll need for any project of any complexity, which are a cumbersome and painful solution to a problem that simply shouldn't exist.
It may technically be possible to write clean and maintainable code in Python if you try hard enough, but you're always skating so close to the edge that eventually somebody is going to get in there tip the whole fragile mess into the abyss.
Also: https://docs.astral.sh/ruff/rules/import-private-name/
The Python ecosystem has many standard tools nowadays to enforce consistent style, including how modules import each other. The ast and libcst modules are very fast and can quickly identify any imported symbols beginning with an underscore:
from a import _naughty
import a
a._naughty()
import a
f = next(
getattr(a, f“_{s}”)
for s in synonyms(“cheeky”)
)
f()
import my_module
# my_module/__init__.py
from .submodule import *
from .another_submodule import *(Besides which, when you do have __all__ just regular wildcard imports work well anyway. I've been using them for >10 years without trouble. I think people just repeat the claim that they're a problem without having 1st-hand experience of using them with __all__ correctly.)
Python's position is that we're all adults here. Don't touch "_Name". Really don't touch "__Name". You can if you're an expert and willing to take responsibility for your actions.
Addendum: And in this ModuleWrapper thing, I can still access `core._module.PrivateApi` so we're back to square one.
Package authors are pretty good about it. But when the only people using your code are your coworkers, people don't seem to put as much thought into having a clean interface.
If you cannot trust your coworkers to respect the _meaning of __names, then how can you trust them with much harder concerns like algorithms and data-structures?
The article contains this amazing quote: """I’ve seen a unicorn startup dump their existing codebase and start over because the modules within it became so tightly coupled together, it was impossible to effectively develop within it or break them apart."""
This quote definitely needs flesh on its bones. I don't think this was Instagram, and I am not sure who else? Could the author chime in maybe?
It's easy to miss an internal api access, so why not make it just impossible?
Obviously mistakes can also happen with guard rails enabled. In a language with explicit exports or private-by-default attributes, you can still make mistakes and expose more implementation details than you wanted. But in my experience, that happens a lot less often than it does in Python, where privacy is just a matter of convention that's easy to forget.
So in that regard, yes I trust myself and my coworkers, even with the difficult parts, but I also assume that we'll all occasionally make mistakes, and if I can find ways to avoid those mistakes, I'm all for it.
Without putting specific companies on blast, I hear about this problem consistently from growth stage startups that scaled on Python. It's a common reason people reach for microservices - trying to use network boundaries to cover for the lack of module boundaries.
Ideally you can trust coworkers; unfortunately that's not always the case. Here's a quote from a developer in response to the article in another forum: """ It can even be very useful sometimes to be able to import private stuff. In a large ecosystem where you use many modules from other teams, you might need to use some private functionality. """ If something is available for an individual developer that solves their problem, often they're not thinking about the fact that it's private or creates a brittle dependency with no contract.
For instance, in C, you have read/write access to the whole program's memory space. Want to work around some security mechanism? A well-aimed pointer will do the right trick.
Python doesn't prevent you from ever touching _internal or __private names. If you're reviewing a Python PR that does that, you should be as suspicious as if you were reviewing C code that stomps all over memory. In either case you almost certainly should not be doing that.
But sometimes, only sometimes, you might be the right person in the right use case to do it. Then you can ignore the linters screaming at you and concentrate on convincing your coworker this is a good idea.