Hacker News Reader

Comment by nine_k | Hacker News Reader

More than that: it seems that Rust stdlib nudges the developer towards using neat APIs at an incorrect level of abstraction, like path-based instead of handle-based file operations. I hope I'm wrong.

View on HN

Nearly every available filesystem API in Rust's stdlib maps one-to-one with a Unix syscall (see Rust's std::fs module [0] for reference -- for example, the `File` struct is just a wrapper around a file descriptor, and its associated methods are essentially just the syscalls you can perform on file descriptors). The only exceptions are a few helper functions like `read_to_string` or `create_dir_all` that perform slightly higher-level operations.

And, yeah, the Unix syscalls are very prone to mistakes like this. For example, Unix's `rename` syscall takes two paths as arguments; you can't rename a file by handle; and so Rust has a `rename` function that takes two paths rather than an associated function on a `File`. Rust exposes path-based APIs where Unix exposes path-based APIs, and file-handle-based APIs where Unix exposes file-handle-based APIs.

So I agree that Rust's stdilb is somewhat mistake prone; not so much because it's being opinionated and "nudg[ing] the developer towards using neat APIs", but because it's so low-level that it's not offering much "safety" in filesystem access over raw syscalls beyond ensuring that you didn't write a buffer overflow.

[0]: https://doc.rust-lang.org/std/fs/index.html

> For example, Unix's `rename` syscall takes two paths as arguments; you can't rename a file by handle

And then there’s renameat(2) which takes two dirfd… and two paths from there, which mostly has all the same issues rename(2) does (and does not even take flags so even O_NOFOLLOW is not available).

I’m not sure what you’d need to make a safe renameat(), maybe a triplet of (dirfd, filefd, name[1]) from the source, (dirfd, name) from the target, and some sort of flag to indicate whether it is allowed to create, overwrite, or both.

As the recent https://blog.sebastianwick.net/posts/how-hard-is-it-to-open-... talks about (just for file but it applies to everything) secure file system interaction is absolutely heinous.

[1]: not path

How about fd of the file you wanna rename, dirfd of the directory you want to open it in, and name of the new file? You could then represent a "rename within the same directory" as: dfd = opendir(...); fd = openat(dfd, "a"); rename2(fd, dfd, "b");

I can't think of a case this API doesn't cover, but maybe there is one.

The file may have been renamed or deleted since the fd was opened, and it might have been legitimate and on purpose, but there’s no way to tell what trying to resolve the fd back to a path will give you.

And you need to do that because nothing precludes having multiple entries to the same inode in the same directory, so you need to know specifically what the source direntry is, and a direntry is just a name in the directory file.

> So I agree that Rust's stdilb is somewhat mistake prone; not so much because it's being opinionated and "nudg[ing] the developer towards using neat APIs", but because it's so low-level that it's not offering much "safety" in filesystem access over raw syscalls beyond ensuring that you didn't write a buffer overflow.

`openat()` and the other `*at()` syscalls are also raw syscalls, which Rust's stdlib chose not to expose. While I can understand that this may not be straight forward for a cross-platform API, I have to disagree with your statement that Rust's stdlib is mistake prone because it's so low-level. It's more mistake prone than POSIX (in some aspects) because it is missing a whole family of low-level syscalls.

They're not missing, Rust just ships them (including openat) as part of the first-party libc crate rather than exposing them directly from libstd. You'll find all the other libc syscalls there as well: https://docs.rs/libc/0.2.186/libc/ . I agree that Rust's stdlib could use some higher-level helper functions to help head off TOCTOU, but it's not as simple as just exposing `openat`, which, in addition to being platform-specific as you say, is also error-prone in its own right.

But those are all unsafe, taking raw strings.

Why can I easily use "*at" functions from Python's stdlib, but not Rust's?

They are much safer against path traversal and symlink attacks.

Working safely with files should not require *const c_char.

This should be fixed .

> But those are all unsafe, taking raw strings.

The parent was asking for access to the C syscall, and C syscalls are unsafe, including in C. You can wrap that syscall in a safe interface if you like, and many have. And to reiterate, I'm all for supporting this pattern in Rust's stdlib itself. But openat itself is a questionable API (I have not yet seen anyone mention that openat2 exists), and if Rust wanted to provide this, it would want to design something distinct.

> Why can I easily use "*at" functions from Python's stdlib, but not Rust's?

I'm not sure you can. The supported pattern appears to involve passing the optional `opener` parameter to `os.open`, but while the example of this shown in the official documentation works on Linux, I just tried it on Windows and it throws a PermissionError exception because AFAIK you can't open directories on Windows.

> AFAIK you can't open directories on Windows.

You can but you have to go through the lower level API: NtCreateFile can open a directory, and you can pass in a RootDirectory handle to following calls to make them handle-relative.

You can open directories using high level win32 APIs. What you need NtCreateFile for is opening files relative to an open directory.

I took parent's message to be asking why the standard library fs primitives don't use `at` functions under the hood, not that they wanted the `at` functions directly exposed.

> which Rust's stdlib chose not to expose

i.e. expose through things like `File::open()`.

> why the standard library fs primitives don't use `at` functions under the hood

In this case it wouldn't seem to make sense to use `at` functions to back the standard file opening interface that Rust presents, because it requires different parameters, so a different API would need to be designed. Someone above mentioned that such an API is being considered for inclusion in libstd in this issue: https://github.com/rust-lang/rust/issues/120426

The nix crate provides the safe wrappers. https://docs.rs/nix/latest/nix/fcntl/fn.openat2.html

The correct comparison is to rustix, not libc, and rustix is not first-party. And even then the rustix API does not encapsulate the operations into structs the same way std::fs and std::io do.

The correct comparison to someone asking for first-party access to a C syscall is to the first-party crate that provides direct bindings to C syscalls. If you're willing to go further afield to third-party crates, you might as well skip rustix's "POSIX-ish" APIs (to quote their documentation) and go directly to the openat crate, which provides a Rust-style API.

If I have to use unsafe just to open a file, I might as well use C. While Rustix is a happy middle that is usually enough and more popular than the open at crate, libc is in the same family as the "*-sys" crate and, generally speaking, it is not intended for direct use outside other FFI crates.

I agree it’d be nice if there were a safe stdlib openat API, but

> If I have to use unsafe just to open a file, I might as well use C.

is a ridiculous exaggeration.

I agree it is an exaggeration in that of course you could write a wrapper. The point was that if everyone had to write their own FFI wrappers, Rust wouldn't go far and openat is not an exception.

There is code available at the right level of abstraction (the rustix or openat crates), and while it's not managed by the Rust team, uutils already have many third party dependencies. Bringing up libc just because it's first party, instead, is comparing apple to oranges.

openat() is there, but it's unstable (because the dirfd-related syscalls are not all fully implemented and tested across all platforms Rust supports yet): https://doc.rust-lang.org/std/fs/struct.Dir.html#method.open...

There are lots of unstable things in Rust that have been unstable for many years, and the intentional segregating of unstable means that it's a nonstarter for most use cases, like libraries. It's unstable because there's significant enough issues that nobody wants to mark it as stable, no matter what those issues are.

As long as it's unstable it's totally fair to say Rust's stdlib does not expose them. You might as well say it's fixed because someone posted a patch on a mailing list somewhere.

There are lots of unstable things in Rust that have been unstable for many years, but this isn't one of them. openat() was added in September, and the next PR in the series implementing unlinkat() and removeat() received a code review three weeks ago and is currently waiting on the author for minor revisions.

> As long as it's unstable it's totally fair to say Rust's stdlib does not expose them. You might as well say it's fixed because someone posted a patch on a mailing list somewhere

Agreed. My comment was intended to be read as "it's planned and being worked on", not "it's available".

After reading this article, I'm inclined to think that the right thing for this project to do is write their own library that wraps the Rust stdlib with a file-handle-based API along with one method to get a file handle from a Path; rewrite the code to use that library rather than rust stdlib methods, and then add a lint check that guards against any use of the Rust standard library file methods anywhere outside of that wrapper.

If that's the right approach, then it would be useful to make that library public as a crate, because writing such hardened code is generally useful. Possibly as a step before inclusion in the rust stdlib itself.

Agreed. (This approach feels like a cousin of Parse, Don't Validate.)

Yeah. The idea is, if you're consistently making mistakes because the most convenient API at your disposal (here, the rust standard library file/directory APIs that are based around Paths), then after you fix the actual bugs you should write a better abstraction and then deliberately add friction around not using that better abstraction to try to constrain future developers (including future-you) from using the more-error-prone abstraction.

Parse, don't validate is also a principle that encourages people to use a less-error-prone abstraction (the parsed data structure or an error representing invalid input), rather than a more-error-prone one (the original untyped data with ad-hoc validations at various call sites).

Unfortunately, it's not the Rust stdlib, it's nearly every stdlib, if not every one. I remember being disappointed when Go came out that it didn't base the os module on openat and friends, and that was how many years ago now? I wasn't really surprised, the *at functions aren't what people expect and probably people would have been screaming about "how weird" the file APIs were in this hypothetical Go continually up to this very day... but it's still the right thing to do. Almost every language makes it very hard to do the right thing with the wrong this so readily available.

I'm hedging on the "almost" only because there are so many languages made by so many developers and if you're building a language in the 2020s it is probably because you've got some sort of strong opinion, so maybe there's one out there that defaults to *at-style file handling in the standard library because some language developer has the strong opinions about this I do. But I don't know of one.

Openat appeared in Linux in 2006 but not in FreeBSD until 2009; go started being developed in 2007. It probably missed the opportunity by a year. It would have been the right thing to change the os module at some point in the last 18 years, however.

https://pkg.go.dev/os#Root

If anything, I find the rust standard library to default to Unix too much for a generic programming language. You need to think very Unixy if you want to program Rust on Windows, unless you're directly importing the Windows crate and foregoing the Rust standard library. If you're writing COBOL style mainframe programs, things become even more forced, though I doubt the overlap between Rust programmers and mainframe programmers that don't use a Unix-like is vanishingly small.

This can also be a pain on microcontrollers sometimes, but there you're free to pretend you're on Unix if you want to.

That's the same for the C or Python standard libraries. The difference is that in C you tend to use the Win32 functions more because they're easily reached for; but Python and Rust are both just as Unixy.

Indeed, though for C it makes sense given its origins, and Python sort of grew from a fun project into a massive ecosystem by accident.

If you want to support file I/O in the standard library, you have to choose _some_ API, and that either is limited to the features common to all platforms, or it covers all features, but call that cannot be supported return errors, or you pick a preferred platform and require all other platforms to try as hard as they can to mimic that.

Almost all languages/standard libraries pick the latter, and many choose UNIX or Linux as the preferred platform, even though its file system API has flaws we’ve known about for decades (example: using file paths too often) or made decisions back in 1970 we probably wouldn’t make today (examples: making file names sequences of bytes; not having a way to encode file types and, because of that, using heuristics to figure out file types. See https://man7.org/linux/man-pages/man1/file.1.html)

You have to choose something, and I'm glad they didn't go with the idiotic Go approach ("every path is a valid UTF-8 string" or we just garble the path at the standard library level"). You can usually abstract away platform weirdness at the implementation level, but programming on non-Unix environments it's more like programming against cygwin.

A standard library for files and paths that lacks things like ACLs and locks is weirdly Unixy for a supposedly modern language. Most systems support ACLs now, though Windows uses them a lot more. On the other hand, the lack of file descriptors/handles is weird from all points of view.

Had Windows been an uncommon target, I would've understood this design, but Windows is still the most common PC operating system in the world by a great margin. Not even considering things like "multile filesystem roots" (drive letters) "that happen to not exist on Linux", or "case insensitive paths (Windows/macOS/some Linux systems)" is a mistake for a supposedly generic language, in my opinion.

As far as I can tell from Microsoft's documentation, WinAPI access for ACLs was added in Windows 10, which Rust 1.0 predates. And std::fs attempts to provide both minimalist and cross-platform APIs, which in practice means (for better or worse) it's the lowest common denominator between Windows and Unix, with the objective being that higher-level libraries can leverage it as a building block. From the documentation for std::fs:

"This module contains basic methods to manipulate the contents of the local filesystem. All methods in this module represent cross-platform filesystem operations. Extra platform-specific functionality can be found in the extension traits of std::os::$platform."

Following its recommendation, if we look at std::os::windows::fs we see an extension trait for setting Windows-specific flags for WinAPI-specific flags, like dwDesiredAccess, dwShareMode, dwFlagsAndAttributes. I'm not a Windows dev but AFAICT we want an API to set lpSecurityAttributes. I don't see an option for that in std::os::windows::fs, likely complicated by the fact that it's a pointer, so acquiring a valid value for that parameter is more involved than just constructing a bitfield like for the aforementioned parameters. But if you think this should be simple, then please propose adding it to std::os::windows::fs; the Rust stdlib adds new APIs all the time in response to demand. (In the meantime, comprehensive Windows support is generally provided by the de-facto standard winapi crate, which provides access to the raw syscall).

> WinAPI access for ACLs was added in Windows 10

I'm not sure which docs you mean but that's not true. The NT kernel has used ACLs long before rust was invented. But it's indeed true that rust adds platform-specific methods based on demand. The trouble with ACLs is it means either creating a large API surface in the standard library to handle them or else presenting a simple interface but having to manage raw pointers (likely using a wrapper type but even then it can't be made totally safe).

> the de-facto standard winapi crate, which provides access to the raw syscall

Since the official Microsoft `windows-sys` crate was released many years ago, the winapi crate has been effectively unmaintained (it accepts security patches but that's it).

As far as I can tell even NFS got ACL support before the first Rust release. NFSv4.1 in 2010 vs Rust in 2012.

> I'm not sure which docs you mean

I was looking at these: https://learn.microsoft.com/en-us/windows/security/identity-...

> the winapi crate has been effectively unmaintained

Shows how much of a Windows dev I am. :P

You'd want to be looking at these[1] instead, especially SetFileSecurity[2].

As noted, the "minimum supported" version means exactly that, and does not reflect when the API function was introduced.

[1]: https://learn.microsoft.com/en-us/windows/win32/secauthz/low...

[2]: https://learn.microsoft.com/en-us/windows/win32/api/winbase/...

You misunderstand the documentation. Microsoft doesn't provide online documentation for versions of Windows that are no longer supported. Functions like SetFileSecurity have existed since Windows NT 3.1 back in 1993.

But the documentation I'm using claims that it applies to Windows 10, which stopped being supported last year.

Windows 10 support is still available to people who pay for it.

SetFileSecurityA is listed as Windows XP+ (https://learn.microsoft.com/en-us/windows/win32/api/winbase/...) but Microsoft has deprecated all pre-XP documentation.

According to https://www.geoffchappell.com/studies/windows/win32/advapi32..., the function was available first in advapi32 version 3.10, which was included in Windows NT 3.10 (14th July 1993): https://www.geoffchappell.com/studies/windows/win32/advapi32...

lpSecurityAttributes just refers to a SecurityAttributes struct (Rust bindings here: https://microsoft.github.io/windows-docs-rs/doc/windows/Win3...) Annoying pointers for sure, but nothing a Rust API can't work around with standard language features.

And sure, Rust could add the entire windows crate to the standard library, but my point is that this isn't just Windows functionality: getfacl/setfacl has been with us for decades but I don't know any standard library that tries to include any kind of ACLs.

> I'm glad they didn't go with the idiotic Go approach ("every path is a valid UTF-8 string" or we just garble the path at the standard library level")

Can you expound a bit on this? I haven't been able to find any articles related to this kind of problem. It's also a bit surprising, given that Go specifically did not make the same choice as Rust to make strings be Unicode / UTF-8 (Go strings are just arrays of bytes, with one minor exception related to iteration using the range syntax).

Go's docs put it like this: Path names are UTF-8-encoded, unrooted, slash-separated sequences of path elements, like “x/y/z”. If you operate on a path that's a non-UTF-8 string, then Go will do... something to make the string work with UTF-8 when passed back to standard file methods, but it likely won't end up operating on the same file.

Rust has OsStr to represent strings like paths, with a lossy/fallible conversion step instead.

Go's approach is fine for 99% of cases, and you're pretty screwed if your application falls for the 1% issue. Go has a lot of those decisions, often to simplify the standard library for most use cases most people usually run into (like their awful, lossy, incomplete conversion between Unix and Windows when it comes to permissions/read-only flags/etc.).

> Path names are UTF-8-encoded, unrooted, slash-separated sequences of path elements, like “x/y/z”

This is only for the "io/fs" package and its generic filesystem abstractions. The "os" package, which always operates on the real filesystem, doesn't actually specify how paths are encoded, nor does its associated helper package "path/filepath".

In practice, non-UTF-8 already wasn't an issue on Unix-like systems, where file paths are natively just byte sequences. You do need to be aware of this possibility to avoid mangling the paths yourself, though. The real problem was Windows, where paths are actually WTF-16, i.e. UTF-16 with unpaired surrogates. Go has addressed this issue by accepting WTF-8 paths since Go 1.21: https://github.com/golang/go/issues/32334#issuecomment-15500...

The `os` package, that is the main way everyone I've seen opens and reads files in Go, doesn't specify any restriction on its path syntax (except that it uses `string`, of course). I've tried using it on Linux with a file name that would be invalid UTF-8 and it works without any issues.

I for one hadn't even heard of the io/fs package that has the problems that you mention, and I don't remember ever seeing it used in an example. I've looked in a code base I help maintain, and the only uses I could find are related to some function type definitions that are used by filepath.WalkDir and filepath.Walk - and those functions explicitly document the fact that they don't use `io/fs` style paths when calling these functions - they don't even respect the path separator format:

  // WalkDir calls fn with paths that use the separator character appropriate
  // for the operating system. This is unlike [io/fs.WalkDir], which always
  // uses slash separated paths.
  func WalkDir(root string, fn fs.WalkDirFunc) error {

Where fs.WalkDirFunc is defined like this:

  type WalkDirFunc func(path string, d DirEntry, err error) error

> Go strings are just arrays of bytes,

https://go.dev/ref/spec#String_types: “A string value is a (possibly empty) sequence of bytes”

https://pkg.go.dev/strings@go1.26.2: “Package strings implements simple functions to manipulate UTF-8 encoded strings.”

So, yes, Go strings are just arrays of bytes in the language, but in the standard library, they’re supposed to be UTF-8 (the documentation isn’t immediately clear on how it handles non-UTF-8 strings).

I think this may be why the OP thinks the Go approach is “every path is a valid UTF-8 string”

That's a norm in most languages, this is just more convenient way to operate

Reply on news.ycombinator.com