Hacker News Reader

Comment by 5e92cb50239222b | Hacker News Reader

Among mainline Linux filesystems, xfs started doing this first. The test suite is still named xfstests, although many more filesystems rely on it now. They regularly test xfs on enormous filesystems which 99.9% of us will never see, both with hundreds of billions of tiny files, and relatively small numbers of very large ones, plus various mixes of the two. Pushing it into edge cases like billions of files in one directory without any nesting. I really like that strong engineering culture and that's why I prefer xfs for most stuff.

View on HN

> edge cases like billions of files

Sometimes edge cases can quickly detect bugs that only happend rarely under normal circumstances and therefore is difficult to reproduce/debug.

E.g. when programming in C for little endian computers it can be a good idea to test code on big endian CPUs as the difference in endianess can reveal "out of bounds" writes for pointers.

These sorts of exercises help with performance tuning in the small.

One thing you should learn, and many don't, about perf analysis is that you start getting serious artifacting in the data for tiny functions that get called an awful lot. I've found a lot of tangible improvements from removing 50% of the calls to a function that the profiler claims takes barely any time. Profilers lie. You have to know what they lie about.

When I'm trying to optimize leaf- or near-leaf-node functions I've been known to wrap the call with a for loop that runs the same operation 10, 100, 1000 times in a row, just so I can see if some change has a barely-double-digit effect on performance. These predictions usually hold up in production.

Just be very, very sure not to commit that for loop.

Or use representative data that is ridiculously large compared to the average case.

This is really not unique to XFS... anyone who worked on a filesystem in at least the last decade would tell you that a test like the one similar to what OP inadvertently created are commonplace.

Unlike with many user-space applications, filesystems have very well-defined range of conditions they have to work in. Eg. every filesystem worth its salt will come with a limit on number of everything in it, i.e. number of files, groups, links and so on. And these limits are tested, they aren't conjectures. Ask any filesystem developer how many metadata operations per second can their program do, and they will likely be able to answer you in their sleep. This might be surprising on the consumer end of the deal, but to the developers there's nothing new here.

Reply on news.ycombinator.com