Comment by steveklabnik | Hacker News Reader

Comment by steveklabnik | Hacker News Reader

Parent Full thread

Hey hey, this post is great! A few small comments:

> It's not obvious to me why they do this – it's a documentation generator, why does it care what version of the Rust compiler I'm using?

rustdoc, being a part of the Rust distribution, has the same stability guarantees that Rust does: once it gains a feature, it will never go away. We (well, not me I don't really work on rustdoc, to be clear) need the capacity to try out new features without committing to them, same as features in the language.

> PyPI launched in 2003, and CPAN has been running since 1995

Neither of these support namespacing though, right?

> Cargo's unit of distribution is the crate, which is a problem because Rust's compilation unit is also the crate.

Cargo's unit of distribution is a package, which is one or more crates. It is true that there can only be one library crate per package, however. This is one area where people don't use the correct words super often, honestly.

> Of course build times are slow if changing one line of a leaf file requires rebuilding dozens of modules.

This shouldn't change based on if you're using Cargo or not; if it's re-building dependencies every time you touch a file, that is a bug.

  > rustdoc, being a part of the Rust distribution,
  > has the same stability guarantees that Rust does:
  > once it gains a feature, it will never go away. We
  > (well, not me I don't really work on rustdoc, to be
  > clear) need the capacity to try out new features
  > without committing to them, same as features in the
  > language.

I guess I expect rustdoc and rustc to be much less tightly coupled, so that I could use (for example) stable rustc and nightly rustdoc on the same code.

In the current model, I can't use nightly rustdoc features without also using nightly rustc, because the annotations are processed by both systems.

You could imagine an alternate model where rustc stabilizes `#[doc(...)]` and leaves everything in there up to rustdoc's interpretation, and nightly rustdoc might enable new parameters to the #[doc] annotation.

  > Neither of these support namespacing though, right?

That's correct. I meant that mostly in terms of "package registries are not a new and untested idea". It's like using strings for file paths: not obviously a bad idea, turns out to be unworkable for obscure reasons, and as a new language Rust got to avoid that particular mistake. It feels like crates.io didn't (/ doesn't) have a similar process of avoiding known mistakes.

  > This shouldn't change based on if you're using Cargo
  > or not; if it's re-building dependencies every time
  > you touch a file, that is a bug.

Given a crate with a hundred files, each one fairly independent of the other (i.e. a broad shallow build graph), changing one requires (if I understand rustc correctly) recompiling them all. Otherwise the `crate::` references can't be type-checked.

> I guess I expect rustdoc and rustc to be much less tightly coupled

Yeah, and that could work in theory. The thing is, rustdoc uses the compiler as a library, so they actually are pretty tightly coupled. This causes a lot of pain but also has a bunch of upsides. I really want to see a rustdoc that is not, but it's gonna take a long time.

> That's correct.

Cool, I thought so, I just wasn't 100% sure! I think it's interesting how different people take away different things here, to me, this demonstrates that you don't have to have namespaces; many of the largest ecosystems do not and have not had them. We did learn a lesson there, just didn't come to the same conclusion that you did. This one is more controversial within the community though, many people do agree with you.

> Given a crate with a hundred files

Yes, but that doesn't change with rustc vs cargo though; it's still a single rustc invocation to compile all those files. (And, it doesn't quite do so, there is some degree of incremental compilation going on, but it's not as fine-grained as it can be. Working on it!)

  > Yes, but that doesn't change with rustc vs cargo
  > though; it's still a single rustc invocation to
  > compile all those files.

When using Bazel instead of Cargo the developer can split the build graph into much smaller crates (down to one per module if desired), and still treat the entire assembly as one package.

I should emphasize that this is a limitation of Cargo, not rustc. rustc does just fine with big libraries, it's only Cargo that makes it difficult to write hundred-crate packages.

Ah, so you're not saying that you take the same project and port it from Cargo to Bazel and it somehow does less work, you're saying that it's easier to make projects with a larger number of smaller crates with Bazel than Cargo? That would be the bit I'm missing, Bazel and Cargo should invoke rustc the same amount of times per crate: once. Cargo doesn't change the compilation model of the language.

(Sorry, I feel a bit dense here, but I am very curious to understand exactly. I haven't used Bazel a ton yet, but am very interested in it.)

Yep, exactly. If I were building rust-fuse with Bazel I would probably split it into about 4 crates (fuse_kernel, fuse_io, protocol, server). This is what I tried to do in Cargo and gave up because of the crates.io packaging requirement.

Gotcha. You can use path as a dependency if you also use the version as a dependency, and it’ll use the path locally but the version when you publish. Maybe that needs to be better documented and would have just fixed your issue. (This is described at https://doc.rust-lang.org/stable/cargo/reference/specifying-... )

That requires publishing the internal crates as separate entries in crates.io, if I understand correctly, which I don't want to do (see the final section on crates.io's packaging model).

Ideally I could have a single crates.io package, with a single tarball, containing multiple crates. The internal crates would not be exposed to users, wouldn't have versions, wouldn't be on docs.rs, and so on.

Yeah, that is true, it does.

I do think that's an interesting idea. I'll have to mull on it. Thanks again. I've been thinking of a few things here, very interesting :)

> Cool, I thought so, I just wasn't 100% sure! I think it's interesting how different people take away different things here, to me, this demonstrates that you don't have to have namespaces; many of the largest ecosystems do not and have not had them. We did learn a lesson there, just didn't come to the same conclusion that you did. This one is more controversial within the community though, many people do agree with you.

Maven is from 2004 and has namespaces. Maven has it's problems, but the Java package management ecosystem gets many things right that later systems have not.

You are correct that Maven is one of the first package registries to use a principled approach toward namespacing.

I didn't mention it because -- and this may be rude to Maven -- I consider Maven's coordinates syntax to be closer to "primitive Go" than it's own distinct thing. If software hosted on `example.com/foo/bar` has the package name `com.example.foo/bar` then it's introducing ambiguity without much benefit.

It's not just the hosting, it's decoupled from where the project is hosted. Some packages do just use an arbitrary groupId. The reversed-domain-name convention makes sure you'll never get two different groups claiming the same name, and aligns with the convention for how code is namespaced in Java, but it is ultimately only a convention.

>The reversed-domain-name convention makes sure you'll never get two different groups claiming the same name

Surely that's only true at any one instant? Domain names change hands all the time.

For example: All non-EU citizens in the UK that have .eu domain will have them forcibly de-registered at the end of the year. What happens to a eu.example.foo package then?

True, but it's hard to imagine a developer accidentally buying a domain that had previously been used by an unrelated developer and wanting to release packages under the same namespace.

Sure, but that’s not really what I was talking about. Citing maven would have made the point stronger for exactly that reason!

(And the lesson isn’t that either, the claim was never “this feature is terrible and ruins the language”, there very much are successful examples. The claim is that the lack of the feature does not mean failure.)

Would you be willing to expand on why using strings for file paths turns out to be unworkable? Very curious what the obscure reasons are!

Short version: file paths are an OS-specific construct, and not all OSes have paths that can be round-tripped through Unicode.

The first bad option is to say "paths are bytes", which is true on traditional POSIX platforms. Then you run your code on macOS and discover it's processing your bytes as UTF-8 and applying Unicode normalization. After fixing that you try porting to Windows and get to learn about `wchar_t`.

The other bad option is to say "paths are (Unicode) strings", which works fine on Windows and macOS. It works 90% of the time on POSIX, but you'll eventually find users with non-UTF8 file paths and then you're in trouble.

My first hand experience with this going wrong is GHC, which started off with raw bytes, then moved to strings using a bad conversion function, then they tried to work around the issue by placing non-UTF8 bytes into a Unicode reserved area.

Java also has problems because it represents paths as strings and tries to use the libc locale for decoding. https://github.com/bazelbuild/bazel/pull/10111 is an example of how this breaks software in ways that are difficult to work around.

Thanks for the response!

If your String type must be valid Unicode (which is true for pretty much every language that isn't C++), it cannot represent all paths. On most unixy filesystems, filenames are arbitrary byte sequences which aren't valid UTF-8, and on windows, NTFS stores filenames as UTF-16, but it allows unpaired surrogates.

What languages require string types to be valid Unicode? Go for example let’s you put anything in a string; it’s only Unicode if you put Unicode into the string including via creating string literals.

I think python also takes the approach where strings must be valid unicode. Both Rust and Python use their String types to represent valid unicode, and something else ([] and byte strings, respectively) to represent other encodings and invalid unicode.

Requiring strings to be valid unicode simplifies the internal workings of (and use of) a language greatly. Dealing with unknown input becomes a single operation at the boundary after which you can safely assume everything is validly printable or processable as text.

I think there is some convenience in being able to use the type system to denote input which has already been validated; however, I can't see how a language runtime benefits from this property with respect to unicode. Notable, printing and processing text aren't operations a runtime typically deals with. Libraries including the standard library might benefit from this property, but even then I wouldn't say the benefit is "great". It's a nice property, but I don't think it makes or breaks anything.

It's worth pointing out that Python 3 thought it important enough to make a backwards incompatible change to separate 'bytes' from 'strings' to effectively get the same result.

Probably, but I think that’s more to do with the typing convenience (which is even more puzzling in Python’s case since it isn’t statically typed).

Rust, Java, Swift...

Is that technically UCS-2 rather than UTF-16?

Technically not since ~2000. Windows uses UTF-16 throughout the OS. That is, Windows strings will be encoded and decoded as UTF-16 not UCS-2.

However the tricky bit is that the kernel doesn't enforce this so it's possible for programmers to intentionally make broken UTF-16 strings. A broken UTF-16 string shouldn't be considered UCS-2 just because it happens to be a valid UCS-2 string (otherwise all bit patterns could be called "UCS-2" so long as they are an even number of bytes in length).

(To add on to the sibling comment's great answer, here's Rust's docs on this https://doc.rust-lang.org/stable/std/path/index.html and https://doc.rust-lang.org/stable/std/ffi/struct.OsString.htm... )

More interesting is what you do use for file paths, if you accept that both strings and bytes don't work. The answer will be I guess a dedicated FilePath type that knows paths are lists of blobs, and the blobs can further be divided up into prefixes and an extensions.

But that doesn't really solve the problem. You still have to convert file paths to strings to print them, and strings to file paths to accept them as from the terminal and databases. For that to work the round trip had better be safe on any given platform otherwise the user can't copy and paste.

Not addressing this problem has been python3's undoing when it is used as a shell scripting replacement. Actually, it's an undoing any time you want to treat "text" as interesting ascii swimming in an unknown encoding, particularly when you want to modify that ascii. HTML with an unknown encoding is a prime example.

You can use #[cfg(doc)] and #[cfg_attr(doc, ...)], but it looks like that still requires #![feature(doc_cfg)], so I guess it wouldn't be compatible with stable rustc anyway, until it's stabilized itself.

> > PyPI launched in 2003, and CPAN has been running since 1995

> Neither of these support namespacing though, right?

But both have policies that prohibit name squatting, whereas crates.io has a policy which explicitly allows it.

Reply on news.ycombinator.com