Has anyone actually found a good use for filenames containing newlines?
Of course, file names are not characters but bytes, and this allows filesystems to transparently support various filename encodings, which are handled by locale, not by filesystem. On the other hand, there are few practical encodings that involve bytes that are control characters in ASCII. Obviously not UTF-8, not 8-bit stuff like ISO-8859-*, nor the widespread Japanese, Korean, or Chinese encodings, as much as I'm aware. UCS2 filenames would be impossible though, but I suspect nobody uses UCS2 as a locale in practice (even ironically).
I think adding a mount flag that would disallow non-printable ASCII bytes in filenames could improve security, and is also really trivial to implement. I cannot be the first person to suggest it. I wonder why it was never implemented.
0 is reserved for the terminator so you can't have a null byte, but I think instead of looking at this from the perspective of "what's the point to support", you should think of it as "why add additional code just for restriction, when we can consider that an upper-layer concern". The same goes for case sensitivity --- it makes the FS code in the kernel more generic and simpler.
Contrast this with the other extreme, filesystems that have to effectively implement a subset of Unicode because they're case-insensitive beyond ASCII. Even Apple abandoned that idea over time: https://news.ycombinator.com/item?id=13953800
Because that way you solve the problem at the root?
Of course that's not the "not my problem" Jersey-style approach.
But tons of our current problems are because of that Jersey-style approach.
Depends on the filesystem. Some expect them to be characters.
They're great for attacking systems, because so many programs aren't prepared for them. Any program that uses filenames as keys (and there are many) can have this vulnerability if the input validation has a weakness.
I don't think there are any legitimate uses. They also make it unnecessarily hard to write robust shell scripts.
That's why things like upload sites usually ignore the provided file name and generate their own (unique) one, or else limit it to a very safe subset like [0-9A-Za-z_]
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
a b c d e f g h i j k l m n o p q r s t u v w x y z
0 1 2 3 4 5 6 7 8 9 . _ -
* The slash `/` is portable as a separator.
* The filename should not be empty
* The filenames `.` and `..` are special. (in many contexts it's reasonable to exclude all components that start with a dot)
* The filename should not start with a `-`
* The entire path should be no more than 256 bytes and no component should be more than 14 bytes (widely ignored since we assume modern filesystems).
I mean it's not an issue in Python for example.
More generally, it's even harder to routinely call programs while connecting them with pipes and ensuring that the programs run in parallel. It's a pain to routinely call other programs and get results Python. It takes many many lines of Python to do tasks that are clearer one-liners in shell... and vice versa.
Python and shell are different; each are good at different things.
> Python and shell are different; each are good at different things.
I agree there. Shell is good for interactive use and throwaway scripts that you don't dare share with anyone.
Python is good for scripts that you want to actually work without supervision. (Though Python is just an example; I would recommend Deno if you are actually scripting stuff.)
Yes - in tests for code that handles filenames, shell or otherwise.
This certainly clarifies that input validation & robustness is a security matter
Though I agree that's the standout quote from the list. I've used it before at work, as appropriate.
Honestly, though, it's one reason I write a lot of my own scripts in zsh these days. Zsh has its own quirks, and I don't touch oh-my-zsh with a barge pole, but it's a darn sight quicker to reason about expansions in zsh code compared to bash, especially if arrays are in the mix. Unfortunately zsh is not necessarily available everywhere and certainly remains outside the base install of many *nix distros.
[1] we can prove the existence of at least one such device, by construction: it's the CI/CD host on which tests run, because those tests themselves may specifically and intentionally be for safe handling of such filenames, and therefore rely on being able to instantiate them
That sounds like a variant of the old broken window fallacy.
There's a lot of ways an arbitrarily named file can get on your system. You could be cloning a random git repo, extracting some sort of archive, or plugging in a usb drive.
I mean... I do expect that on Linux because shitty Bash scripts are so common. But that is a problem that should be solved.
Yes. They are great for more legible names for document files on the Desktop, or even for using as post-it notes, also on Desktop. Of course, it only works for people who don't start fullscreen tmux session as the first thing they do after the boot and then never leave it.
If you have a good reason to create a file, and the value you want to identify it by contains a newline, then that is a good reason to create a filename containing new lines. Filenames are arbitrary bytes excluding null. Regardless at scale, eventually such a file will end up being created. Encoding bugs, libraries, user creation, messed up copy/paste, etc. File names may not even be utf8.
`ls | cmd` is also just an sh code smell. Bash is big on discipline and suffering and not so much on correctness (like C and forth) so code smells matter more. When I worked with/maintained large bash/sh scripts, one `ls | cmd` and I would reread everything the person wrote because it was very likely it was horribly broken. Another one is 'set -euo pipefail`, which people love to set but often don't understand.
A lot of sh/bash, maybe most of it, is written by people that don't take it seriously as a language, and it shows. Also basically everything on stack overflow about it is wrong, and probably everything chatgpt says. I remember it being uniquely bad in that regard. To the point that whenever someone says a neat bash fact, it is better to assume they are part of an elaborate conspiracy to get you to write catastrophic bugs until verified independently. Anything short of that level of paranoia and you eventually do the steam thing and delete everyone's files.
The article could have been titled "why you shouldn't use sh".
Sure, but practicing defense in depth is often a good idea. Maybe today's version of your application sanitizes user input correctly, but tomorrow's version might suddenly allow funky characters. If your super duper bash backup script doesn't get updated, you might suddenly have a problem.
The reason it’s not the least of your worries is if somebody can control a file like that the next step is some kind of escalation and this right here is one of those escalations.
I don't think that's how /tmp works. Aren't those files garbage collected either at boot time or after X days of creation? I mean, if your program creates a file in /tmp and keeps it open for days then you clearly screwed up as you're misusing the whole concept of a temporary file. For that there are better places in the file system hierarchy.