Example: ./jasondir/aa/bb/cc/aabbccdd
You won't like what happens when you put 100k files in one directory.
You can also run into issues like "Argument list too long". ARG_MAX is larger on linux than it used to be, but it's pretty short on older kernels. I assume similar issues might exist on other operating systems.
mkdir uuid; cd uuid
uuid -v 4 -n 1000000 |\
while read uuid; do
touch $uuid;
done
time ls uuid|wc -l
425621
real 0m1.796s
user 0m1.552s
sys 0m0.240s
This is in a hyper-v vm on a Surface 4 pro/i5.
The fact that the "uuid" program can quickly generate uuids make me wonder if maybe one approach would be to generate uuids to (a) fifo(s), and then let db thread/processes read uuids from the other end?
Even at 400k files, you see longer wait times if you've aliased ls to ls --color (pretty common), or use something like ls -F. Either runs stat() on every file.
Then, somewhere in the 1m+ range, it gets unusable.
As this is a standard Ubuntu install, ls is indeed aliased to "ls --color" - but afaik ls as standard detects pipes, and turns off color (so you don't get a lot of control characters if you do "ls --color | sort > file.txt". Unless you use --color=always if I recall correctly.
Just wanted to differentiate between "select * from documents" and "select count(*) from documents" being slow.