Clear explanation of Rust’s module system
sheshbabu.com
sheshbabu.com
To all others struggling with Rust's module system, it may seem unnatural if you're coming from another language (like python), but once you finally understand how it's wired it will be as simple as python's.
Consider `mod.rs` like `__init__.py`, with the added benefit of being capable of renaming and choosing when to expose internal module components.
Yes, didn't realize it would be confusing. I've fixed the diagrams in examples 2 & 3 to include mod.rs in filesystem tree :)
That's exactly what __init__.py does too, though.
Some people learn that you push the stalk up to indicate your intent to turn left, or down to turn right. And then they might eventually learn that in a European car, you push the stalk down to turn left, and up to turn right. And then of course some of these people get confused when switching between different cars. Yes, I've seen plenty of people on the road who indicate one direction and then turn another. It's... kind of frightening.
The other way of learning to use car indicators is much simpler: push the stalk in the direction that you would turn the wheel. Of course this still leaves the possibility that some people will push the wrong stalk, and briefly activate their windshield wipers. But it's a much easier mistake to _notice_, and the consequences are minor.
Back to Rust's modules. I keep seeing articles trying to offer a simple or clear explanation of how they work, that end up unnecessarily complicated in a way that feels a lot like the "up/down" model of car indicators. The explanation that makes the basics of Rust modules clear to me is this:
- Child modules must always be _declared_ in the parent module, or they don't exist.
- The content of child modules may be _defined_ either inline in the parent as `mod child { ... }`, or in a file with a relative path of './child.rs' or './child/mod.rs'.
Did I miss anything important?
Without this basic explanation up-front, I have no idea what to do with the stream of information I'm reading in a lengthy article on the topic -- I've been given no scaffolding onto which to bolt all the details and examples. So this is the "bottom line" that I would like to see "up front" in descriptions of Rust's modules.
Other misunderstandings, e.g., around item visibility, are explained really well by the compiler if you get them mixed up, so I'm not sure how much value there is in mixing them in to articles about how modules are _structured_ before those two basic facts are presented.
I was very confused for a good moment, because the turn signal in my car (which I drive in California) is on the left side of the steering wheel (itself on the left side of the car). I didn't realize the stalk might be on the right side of the wheel in other driver's-side-left cars until you said "some people will push the wrong stalk".
The "push the stalk in the direction that you would turn the wheel" method is intuitive to me.
I think you’ll lose beginners at “These are the different ways we should be able to consume our modules”.
There is no description of what a module is, why they’re needed (or are they optional?), and what that “Cargo.toml” file is about.
I happen to know (rightfully or wrongfully) Cargo is Rust’s package manager, but does Cargo require modules (I would guess so, but reading https://doc.rust-lang.org/book/ch07-01-packages-and-crates.h..., it may need packages, but not necessarily modules), do modules require the use of Cargo (I would guess not, but then, why make a discussion of modules harder by adding Cargo-related stuff?)?
Modules, understood as such, are inevitably connected to the stuff that is happening "one level above" where these things providing the modules are installed (Rust crates, npm packages).
Reaching for a language-independent word, I'd propose to call these things "components" (slightly longer-form thoughts here: https://blog.burakemir.ch/2020/07/modules-components-and-the...)
... but those things happen to be called (or heavily associated with) "module" in Java, C++20 and go.
It is very unlikely, though, given that means there are already two slight variants. Introducing another has a very high cost.
It is normal to have a set of tools that you like or wish to retain. There may be a package installer (rpm, apt, msi, etc.), a build system (plain Makefile, autoconf, CMake, the Visual Studio project/solution files, etc.), a linker (link.exe, the old binutils ld, gold linker, etc.) and so on.
Most of us aren't looking to replace any of that. Even if we are, the stuff coming along with Rust is probably not what we are looking for.
Cargo is thus, to many of us, an unwelcome surprise.
I come from Python, and it would be ridiculous if I said "well we already have Poetry so why doesn't Rust just use that". I don't think saying the same thing about the C++ tooling is different.
Rust is a natively-compiled systems language that supports the C language ABI.
Python, Perl, Java, Lua, Ruby, Scheme, and other such things are expected to bring along some extra gunk. I'm not saying that this is good, but at least it is unsurprising.
Rust needs to fit in with C, C++, Ada, Fortran, Pascal, and all the other natively-compiled systems languages. Using generic system-provided tools isn't just for C. You'd likely use a Makefile to compile Fortran.
The idea is that the languages are interchangable parts with compatibility. You can create a project that uses a different systems language for each source file, run "make", and it all builds. There is no glue code, no interface definition language, no wrapper, no thunk, or anything of the sort.
The expectation is just different. Nobody expects Python to cooperate with Lua or Ruby or Perl, with source files in different languages freely calling into each other. That is the expectation for natively-compiled systems languages.
Also of note, aside from finding the crate root and dependent crates (these are passed explicitly to the compiler, from Cargo most of the time), all of this is strictly part of the Rust language, and handled by the compiler. I find the language model really clear and explicit. I would not touch it, aside from discouraging the use of mod.rs.
I think there's room for an article that expands into Cargo's convention-based source model, which is a bit trickier. A package directory can contain multiple crates: at most one library crate rooted in `src/lib.rs`, but also the various executables that are discovered from the filesystem: src/bin/*.rs, src/main.rs, examples, the test binary, bench binaries… These are crates of their own, with examples and benches needing a library crate to be enabled to access files in the src/ folder (without ugly hacks).
I am glad it exists though!
The only step in the logic that isn't explicitly explained is why you need to do 'pub mod' as supposed to just mod.
From the examples, it seems like there is exactly a 1:1 mapping with the file system, and it's only the visibility of that mapping that is changed by the `mod` keyword. Am I missing something?
> We declare a file as module in its parent, not in itself
So I need to declare the `models` module in `main.rs`, even though it's only used by the `routes` module and not `main.rs` itself? To me, that seems like it's breaking modularity.
Instead, the author points out that the module system ignores undeclared source files.
There are a couple of practical reasons that prevented us from automatically deriving the module tree, as part of the 2018 edition changes to the module system:
* Interaction with partial, in-progress programs. People wanted to be able to comment out `mod` declarations to temporarily exclude files from the build.
* Interaction with version control, e.g. `git stash`. Similar to the above, people want to exclude untracked files from the build when their corresponding `mod` declarations are stashed.
Notably, this is basically the same set of problems people face when using globs in their build system. Maintaining an explicit list of source files can be slightly tedious, but it avoids a lot of surprising behavior in the edge cases around removing files. Rust's list just happens to be distributed throughout the source files, rather than consolidated in a build system file.
They aren't though. Even if the module isn't `use`d, it's still compiled.
(And it has to be. If nothing else, there could be a trait impl buried in the module, which gets shared without `use`.)
Why would this be the case? Under this model, are we not allowed to have public symbols anywhere in a library but at the root? In other words, why should I have to "use" something from a module from elsewhere in this crate to get it to build?
I'm already hopelessly confused by your system. The current model is simple, correct, and efficient. No wonder people hate it so much - it's human nature to hate the unfamiliar!
That messy state may include a new, partially-implemented module file. By default `git stash` will leave it alone, because it's untracked. But `git stash` will also remove its corresponding `mod` declaration, and any uses of it, all at once.
Now say you want to build and run. If that WIP module file gets picked up automatically, it will probably break the build! It may simply be incomplete, or it may depend on some of those stashed changes, or whatever else. Even if it happens to typecheck, now you're assuming that it's all dead code- but it may have some public API (including trait impls!), and still be included in the build!
In fact, we already have a taste of this kind of problem, ever since Cargo started running build.rs scripts automatically, even without a `package.build` key in `Cargo.toml`. If you add a build script with some build-dependencies, run `git stash`, and then build, the build script will fail to compile!
Perhaps you could argue that this is a deficiency in `git stash` that Rust shouldn't have to be concerned with. But if `git stash` removed untracked files as well, that would simply break other use cases instead! For example, I often have a few extra files laying around, which I don't ever intend to commit or .gitignore- sample inputs or outputs, short-term todo lists, or other scratch-pad-like data. Ideally `git stash` would leave those alone- but it's hard for git to tell those apart from the problematic cases above.
That is, my own personal workflow poses essentially zero risk of deleting or committing those files, so "quit using the file system the way I've always used it" is no less of a stumbling block for me than "maintain `mod` declarations" is for you.
We simply haven't found an approach that doesn't step on someone's toes, so the one that Rust was already using from the 2015 edition stuck.
Except the current system actually has the opposite effect -- working with partial programs is actually worse from the standpoint of using rust analyzer. Newbies like myself start writing a new source file and expect that completion and other features will work right away.
> Interaction with version control, e.g. `git stash`. Similar to the above, people want to exclude untracked files from the build when their corresponding `mod` declarations are stashed.
They can just stash the file as well.
This strikes me as making the whole system harder for new people to understand just to smooth over edge cases for power users that definitely have the ability to come up with workarounds.
This is exactly what it is, and due to inertia it won't ever change. At this point, there is no other choice but to either accept it or find peace in smth else.
The 2018 module system was already a substantial redesign, and because of that, we chose to carefully minimize churn to existing programs (which involved two iterations on the design before the 2018 edition released); the vast majority of programs designed for Rust 2015 will Just Work with the 2018 module system.
A future edition could very carefully allow for filesystem-based modules, and focus primarily on that without having to deal with other module system changes at the same time. I can imagine some possibilities that might achieve that, if someone has the bandwidth and diligence to work on it and test it.
Just to be clear, I have the highest respect fo the Rust core team, I've followed some discussions and you guys are going above and beyond carefully weighing all the pros and cons, often to an exhaustive degree.
I feel the main strength of Rust is also its biggest weakness: alway striving for the perfect, platonic ideal solution, and going the extra mile to get there, which then might discard a much simpler but slightly less powerful solution. Often thats a good thing, not sure it was warranted for the module system. I remember there were different opinions in the core team, Aaron being a proponent of the simple version I think.
At a minimum, this could be an off-by-default configuration. Ideally, it would suppress errors caused by these extra files, and downgrade errors caused by missing `mod` declarations to warnings, with a quick-fix to add the declaration.
> They can just stash the file as well.
This is an option, but it's not clear-cut whether it's a better one. See my other comment: https://news.ycombinator.com/item?id=23890842
> This strikes me as making the whole system harder for new people to understand just to smooth over edge cases for power users that definitely have the ability to come up with workarounds.
This certainly was not the goal. The goal of the 2018 module system changes in general was to make the module system easier to learn than it was before; this aspect was simply retained as-is because we couldn't figure out how to make it better without making existing aspects worse.
"There's more than one place you have to write it."