Using the Linux kernel's Case-insensitive feature in Ext4
collabora.com
collabora.com
> There's some excuse for case insensitivity in a legacy model ("We didn't know better"). But people who think unicode equivalency comparisons are a good idea in a filesystem shouldn't be allowed to play in that space. Give them some paste, and let them sit in a corner eating it. They'll be happy, and they won't be messing up your system.
Linus Torvalds, 2014, https://web.archive.org/web/20150112214037/https://plus.goog...
But that being said, are many nontechnical users typing filenames anyways?
That doesn't mean they won't still have trouble working with filenames that differ only by case, and it can still be pretty hard to explain why two characters that are visually similar are distinct to the computer.
But the idea that letters that look different are different is not a hard one to understand. It's the default, and one that people have to actively unlearn.
“Report (final).docx” vs “Report (Final).docx”
Case insensitive comparison is less well defined than you might think. You need to know what language, because the rules are different in different languages. You need to know what version of unicode, because the rules can change. You need to know what to do if the data on disk doesn't conform to the rules. You need to know what to do when copying data from one filesystem to another when the names can't be stored on the destination.
I feel it's often better to train the user on how to use a simple system effectively, than to build a complex system. You have to put the complexity somewhere, and humans are easier to change than software and hardware.
You already need to know that, because filesystems are a jungle. This covers everything from special characters, to non-Unicode encodings, to different Unicode-like encodings, to special names, to file name length limits, and path length limits.
Consider that on Linux a filename is just an arbitrary sequence of bytes with some limitations, and on Windows it's an arbitrary sequence of 16-bit integers. You can put unpaired surrogate characters in a Windows path, and you can put 0xff in a Linux path.
> I feel it's often better to train the user on how to use a simple system effectively, than to build a complex system. You have to put the complexity somewhere, and humans are easier to change than software and hardware.
Filenames are something that affects everyone who uses computers for more than just browsing the web. By and large, these people have been using case-insensitive filesystems for the past 40 years, except the people using Unix-derivatives other than macOS--a small slice of people with a high percentage of experts.
So if you have an idea of how to train a billion people to deal with case sensitivity problems that they didn't have to do with before, I'm all ears.
And yes... the case folding and locale problems exist. I think we're already on the way to the right solution there... hide the filesystem completely from most people who use computers. The filesystem is a terrible way to organize data, so we are building more complicated filesystems and more complicated stacks on top of filesystems to make it easier for humans to work with, and rightly so.
I don't think these are as equivalent as you imply. While it's technically possible for Windows paths to contain unpaired surrogates, in practice it never happens outside of malicious software (and even that's rare),
Whereas on Linux, getting a "random" bag of bytes is relatively common because the convention of UTF-8 is new (and is a userspace convention). Of course it's usually not truly random but you have to play guess the encoding if you want to interpret it.
It's a bit harder for the people who had case insensitivity before, I'd imagine. But the basic way to train people for this is a) to work consistently and b) drop helpful examples in places. For a) When someone makes a new directory lowercse dotless i when they already have a folder named I, they might be surprised to have two directories with the same letter, but they'll figure it out. The same way that they figured out news.ycombinator.com/REPLY doesn't work, because the bytes aren't the same.
For b) make some tools expect a file called Makefile and not work if makefile is present. Done!
Computers are for end-users. OS and app developers are in the minority, and we should remember that when we write apps or filesystems.
Why is this the exception?
For me case insensitive filenames are horrible UX, for my parents case sensitive filenames are horrible UX.
However I wouldn't advice my parents to use case insensitive filenames: there are too much complications that you might get from it in interaction with literally anything. The UX net effect would be negative for them.
That aside, inferring semantic meaning based on a few cultures at this moment in time is a dumb choice. There are tons of such language-specific semantics that could be implemented and the result is needing to memorize all those rules about when it does and doesn't actually matter (and arguing that only certain cultures and languages should have their own semantics encoded in the OS is its own problem).
The principal question is whether the tool is more important, or the end user. Why did I pay for this machine if it weren't intended to facilitate me? That's the bottom line with most of these kinds of technical "correctness" arguments
And as for whether userspace should catch up, thanks to OS X for the most part that already happened a long time ago for a ton of open source packages
I also side with this view, namely that this is something that would be better placed in the userspace rather than the kernel, which really doesn't need more complexity for things that add so little value (negative value to some).
|I've looked at the source and there are pieces that are good and pieces that are not. A whole bunch of random people have contributed to this source, and the quality varies drastically
I happen to agree with the idea that the filename should be a dumb blob of bytes and the kernel should not do case folding, as it is the wrong layer for that, eg. the user can change their language but it won't update what has been written to the disk in thousands or millions of places where you could suddenly have a filename collision somewhere based on those rules changing.
But, I do hope you get that refund for your Linux.
Well, now your filename is invalid utf8. How should programs display it or even address such a file?
what's wrong with foo����.txt
> or even address such a file? ... by using the array of bytes ?
That's reality. An OS which can't keep up with reality is broken.
This hasn't been true since the days of CP/M
All of the wide/special-case manipulation when writing code on e.g. Windows drove me nuts.
It also means that if you want to do something highly annoying and unusual, you can – and arguably case-insensitive filesystems are a subset of that.
I've been running this for years accessing them both from Linux desktops and Windows desktops, and only once have I had an issue that required me to manually rename something on the NAS.
This makes sense as most applications don't care about the filename, and will just use what you supply, or generate one and use that string all over.
Very costly, but the only way to give case-insensitivity.
That is very costly — but it's certainly not the only way to provide case-insensitivity, nor is it the recommended way, and I'd be surprised if any implementation of case-insensitivity actually did what you say.
Normally one would lowercase (or uppercase) both strings, and then do the comparison.
The complexity here usually comes when case-folding various tricky locales.
Windows slient sends a name "foo" over SMB2+. Samba does a stat("foo", &st) call, gets ENOENT.
Now, does that name really not exist ? Or is it there in the requested directory under any of the names "FOO"/"FOo"/"Foo"/"fOo"/"fOO"/"FoO" etc. ?
The only way for an application to tell is to do:
opendir() fname = readdir().... check strlower(client_provided_name, fname); closedir().
If the file really doesn't exist you have ended up scanning the whole directory.
Because Linux doesn't provide directory leasing you have to do this for every missed lookup (another process might have created it in the meantime).
There is no POSIX API for "does this file exist under another cased name" ? If there were Samba would use it.
I now understand you mean that the directory is in fact scanned (and filenames compared in a case insensitive manner) once a miss occurs. Which is what I was hinting at.
Apologies, I misunderstood your post — and my reply could have been clearer also.
The problem is emergent behavior. We can create any number of features, but we have a really hard time testing all the available configurations that result. Engineers rely on simplicity as a way of warding off this particular problem, because the other bottom line is people don't want to pay for a system that loses or invalidates the work they've put into it.
The only real purpose of this is for easier/faster compatibility with software developed and tested on macOS and Windows, which has accidental case inconsistency in file name references in the code, which happens to work fine on macOS and Windows.
TFA could have made that argument, but it didn't, it made incorrect arguments instead. For example, a user might type in a lower-case name for a file one time, and a capitalized name for a file another time, and intend to access the same file. But what user is typing in a whole filename the second time? They're picking it from a list, or if they're very advanced, using completion in a terminal. TFA also mentions non-English languages, in the context of unicode normalization ... but non-western-european languages won't be handled correctly by any universal case-folding algorithm anyway, with Turkish being the most common example.
The whole strategy just doesn't work out well. Many low-level filesystem developers have known this for over 20 years. It doesn't work better for anyone, but non-technical people just aren't aware of why or how it increases complexity and costs, and reduces performance and reliability.
Move on to most anywhere else and the notion that it's fast, reliable, and safe to convert case gets lost in the realities of human writing systems.
And even then it still doesn't fix the issue outside of the world of ASCII. A filename written in hiragana and katakana is logically the same to the end-user, but they are still distinct. Simplified and traditional Chinese, Hangul and romaja, pinyin, devanagari, thai, and the list goes on.
Case-insensitive filenames fixes nothing, but breaks everything. There is only one sensible thing to do with arbitrary user-input, and that is to leave it be.
I wish programmers would believe this about names and addresses! My wife has a two-word first name, and I have a two-word family name.
It must be in the kernel. Else implementations of stuff with human-derived semantics should move out of the kernel, but moving SMB into userspace would of course be ridiculous
So it complicates any mapping of filenames to data structures in the kernel (all 3 of them?), big deal. Every popular desktop operating system supports it, and basically the only reason Linux does not is a mixture of FUD and fear that we may need to update the code at some point due to changes in human culture, the horror!
Meanwhile, typing "cd music" in a terminal need not print "Command not found" when there is clearly a folder named "Music", like the $3k worth of gear in front of me had the complexity of some 1950s b-movie scifi robot
Case-insensitive path lookups on SMB happens in the server (usually samba), in user-space. The client is also usually in user-space through FUSE or the client libs, but CIFS of course exists as a kernel-mode alternative.
And as I have written elsewhere, sure, "music" vs. "Music" is simple when you live in an ASCII world. Trying to be smart with user input only causes trouble for the rest of us. Hiragana and katakana is also logically the same, and on a kana keyboard a similar typo. Simplified and traditional chinese is also logically the same.
> Every popular desktop operating system supports it
Doesn't mean we should break stuff here as well.
SMB in the kernel is a rather dangerous game. Not saying it can't be done (and I know Samsung is doing it :-), but it's a significantly more complex protocol than even NFSv4 (which also really shouldn't be in the kernel either). For complexities sake, userspace is easier IMHO (much easier to debug).
Also, you seem to be of the opinion that code in the kernel must be "magic", in that it can do things that user-space can't. The "missing case" lookup problem still exists for kernel code as it does for user-space code.
Looking up "foo" in all case variations in a file containing a few million files still has to be done by search in the kernel as it does in user-space. It's going to be slow there too unless you provide a case-insensitive indexing mechanism.
Now the kernel offers easier opportunities to do things like directory content caching, which currently aren't exposed to user space via any API - but once you have to do something like directory content caching it's also possible to expose that feature to userspace via an API.
> Else implementations of stuff with human-derived semantics should move out of the kernel, but moving SMB into userspace would of course be ridiculous
Samba would beg to disagree of course :-).
In practice the case insensitive thing if we want to call it should be implemented more high level, in the file manager, rather than in the filesystem itself. That is even what newer versions of Windows/NTFS do! Recent versions of NTFS are in fact case sensitive, and if you mount a NTFS volume with Linux in fact you can create two file with the name differing only by the case: the whole case-insensitive thing is handled at an higher level in the Windows APIs.
Now, if this needs to be taken care of at the application level, then why have this misfeature? It'd be better to have a good library for matching this that could be aware of the language and locale (or maybe multiple lang,locale pairs) instead of throwing this feature into the fs and call it a day.
Also, if some applications benefit from having insensitive matching, like things built to run on Windows/Mac, then having a wrapper that fixed the fs access with this matching library would be enough, no need to force other applications to use insensitive matching because a single one needs it.
That’s a bold claim. Open source developer here (on the fish team): given a user-entered path, how do you make it canonical?
Specifically, without recursively opening directories and listing contents, how can you convert /foo/baR to the correct case as it appears in the filesystem index? Without following any symlinks, should any of the path components be a symlink?
(meaning that filename handling with the same program isn't even consistent with itself)
Also, what happens when you backup very-important-file.txt and VeryImportantFile.txt and then restore to a case-insensitive filesystem? I'll tell you - unexpected things.
Both files get restored as independent files, as case isnt the only difference in those files.
(Did you mean very-important-file.txt and Very-Important-File.txt?)
oh sorry, ha! yes.
By the way, having two files in one directory that would clash on a case insensitive filesystem is a bad idea and pretty unlikely.
It's easier to keep things simple, and rely on the program to discard things that it can't or doesn't want to use. If a program wants to insist that "Bob.txt" and Robert.txt" are the same file, that's its prerogative, although it's a dangerous assumption.
If you can, then how you choose to interpret those bytes is irrelevant.
If you cannot, then you must learn when multiple bytes have the same meaning or when a byte in a different context should have a different meaning.
The latter case is much more complex for the computer as it requires parsing instead of comparison and is more complex for the programmer as it requires understanding of every supported difference (plus learning new ones as they are added).
For instance, take a file named "ivory.txt": `stat("IVORY.TXT")` on a case-insensitive filesystem would succeed if the locale is en_US but fail on tr_TR due to the uppercase version of 'i' being 'İ' there instead.
99% of the time I use lower-case only filenames, as they are easier to read. The times I don't shell completion and/or GUI selection obviate the need to care anyway. Given the significant complexity of Unicode I'm not sure insensitive is the way of the future.
There was a window of time where insensitive made the most sense, the time of DOS. 8-bit per character filenames, with a very primitive CLI shell, ie. no assistance. Now? In the days when a majority of users don't even see filenames? Meh.
Very interesting feature, of course.
İdata.db
Idata.db
What about if only the first file exists, and you open for writing a file called "Idata.db"? Should it overwrite the first file, or create the second?
If it should overwrite the first file, unpacking an archive that contains both will result in a missing file.
What about these two, you might feel differently about because they are visually distinct:
ßdata.db
SSDATA.DB
In some environments, the answer doesn't change but doesn't match what happens in other environments. E.g. many systems implement ASCII case-folding only, others implement more characters but not Unicode, others implement all of Unicode.
Inevitably a program somewhere will break because this wasn't anticipated.
A backup using an archiver will not restore what was there before, and something that worked before the backup will behave differently after the restore.
A script (such as a build script) may inadvertantly create both files when run with one language setting, then break when the language setting is changed, even thought the script doesn't have anything to do with languages.
Your carefully written novel, that has chapter files with names you thought were distinct, will have missing chapters when you copy it across to another system. You discover this months later when you start working again on the novel, long after destroying the original system you thought you had copied from.
It introduces near-endless complications, especially in the face of Unicode, and one should think thoroughly about it before implementing it, as you can read in Torvalds' initial reaction linked elsewhere in this discussion. But the way they did it seems relatively sane to me, with an easy way to compartmentalize it to, e.g., a Steam directory.
I don't really have a use case for that feature right now. But that doesn't mean others don't have a use-case for this feature.
As the comment section reads, most feel entitled to NOT HAVE this convenience feature implemented for others, because they do not need it. It feels like a cognitive bias, but does it have a name, besides "get off my lawn?"
$ ls -l /usr/local/bin/virtualbox
-rwxr-xr-x 1 root wheel 77 Oct 2 2015 /usr/local/bin/virtualbox*
$ ls /usr/local/bin | grep virtualbox
<nothing, WTF??>
$ ls /usr/local/bin | grep -i virtualbox
VirtualBox*
My coworker uses MacOS, I use Linux. Several times, changing the case of a directory or file caused the MacOS to mess up the local git repo so badly that we had to blow it away and refetch from remote.
At least the Mac Finder allows changing the case of a filename. Windows File Explorer simply refuses. I change the case of the file name, add a random character, save the file name. Then edit the file name again, remove the random character, save the file name. It's needlessly annoying.
I think case-INsensitive makes more sense for software developers, since there are many instances where things like "string.h" is not the same as "String.h". But for normal people, case-INsensitve may be more useful. However, even for normal users, I think there are situations where "bob.html" (a noun) is different from "Bob.html" (a person).
Just tested on Windows 10 and this is definitely not true. I could have sworn you could always do it, but my memory of Windows < 10 is foggy. And you can also do the renaming from any console app (cmd, powershell, etc), FYI.
NTFS is case-preserving, but case-insensitive. That, IMO is a good balance. When they were first making NT with the posix subsystem, I suppose they found that the number of cases where two files that only differed in case lived in the same directory were small enough to not matter.
I personally don't think case-sensitivity in file names adds anything useful.
It’s interesting that Microsoft effectively added the reverse feature to NTFS [1] to support per-directory case sensitive files to be more compatible with Linux.
[1] https://www.tiraniddo.dev/2019/02/ntfs-case-sensitivity-on-w...
Here is a user complaint about this problem from Nov 2019: https://answers.microsoft.com/en-us/windows/forum/all/cant-r...
Here's another post from May 2018, explicitly mentioning Windows 10: https://answers.microsoft.com/en-us/windows/forum/all/why-ca...
But I checked again on my own Windows 10 machine (Windows 10 Pro, Version 2004, build 19041.450) and holy shit, it now works. It must have been fixed in a recent Window 10 update.
But making it so that it requires a kernel compile time option to make your filesystem mountable is a level of absurdity that i would not have believed possible before 2020 made unbelievable the new normal.
I have never seen anyone at all mope over lack of the latter, and the former seems quite specific to get such a big feature landed over. What other use-cases are there that benefit from this being in the FS?
Moreover, as complex as the problem can be, it might come in handy to have a unicode normalization subsystem in the kernel for other modules to use. I don't think it's in the FS, but it's now a feature of the kernel that various FS can choose to wire up with, and expose knobs to the user to leverage it.
Besides that, whether they do or not is their choice. But I believe this subsystem is going to get used a lot more than some other, obscure kernel areas ;)
Case-sensitivity is a classic case of users being forced to comply with the computer's needs, rather than the other way around. I contend that that is Wrong, period.
To put it another way: Unicode is the opposite. It is computers adopting complexity to serve the needs of humans. If we can do unicode, we can do case-insensitive filename matching. If we're going to insist on case-sensitivity and ignore the needs of human language, we should just go back to plain old ASCII and force the humans to comply with that too.
With hidden filenames on mobile, desktop GUIs, terminal CLI completion, and shells with insensitive matching like fish (bash via option), the benefits of insensitive fs are not as high as they used to be.
Meanwhile, the complexity of matching modern Unicode causes performance degradation and exposes many edge cases.
In short, the window of time where insensitive made sense has largely closed.
We've got better at working around case sensitivity, sure. I don't see that as a good reason not to fix the problem properly.
As mentioned, there is no way to "fix the problem properly" and a close-enough solution, rarely needed these days, slows performance.
I'd guess that the f2fs changes at least are motivated by sdcard emulation on Android: sdcards were FAT32 in the beginning, so once they started being emulated with a backing store on the regular (ext4, then f2fs on pixels) data partition, case-insensitivity had to be emulated (first via a FUSE daemon in userspace, then via sdcardfs, a stackable FS in the kernel written by Samsung).
(I work at Google on Android, but I don't have any first-hand knowledge about this.)
We got it working fairly well. We expected that we'd have to put in some kind of hack to deal with filename case, and I think we eventually did.
But before that, when it was still just passing Linux case-sensitive filenames through to the case-insensitive Windows filesystem, I tried installing most of whatever was the current release of Red Hat at the time.
It almost all worked fine. The only thing I remember being a problem was that some things, such as some Perl modules from CPAN, had both "makefile" and "Makefile" in the same directory.
For example, common Unix convention in the 1980s-1990s was to name user-created directories with a capital to make them easier to see in a regular directory listing without color (e.g. Poems is a directory, while poems is just a file).
I've used it heavily with content revision (e.g. CATHENA02.yaml is the second cathena configuration file in testing while cathena02.yaml is the production version of it).
Plus, making a filename case-insensitive for processing purposes in a scripting language is very easy.
Consequently, I can't imagine a reason why I'd use case-insensitivity in ext4.