> I finally gave up and wrote a multi-threaded c++ program to "inventory" every file and then I imported that into sqlite and started building out scripts to do bulk actions, set a flag on for files to delete and some scripting to show the files that would be deleted etc.
I did something not dissimilar. Basically at some point, years ago, I got fed up of deleting for the ten times that one picture I already had deleted in the past.
So I created a little database of files that are "tagged", based on their size first, then on their hash.
The "tag" is basically: "File with this size and hash can always be deleted, no matter what". Or "File with this size and hash and this name can always be renamed to this-or-that-name".
I've also got a few rules: "Empty directory with that name can always be deleted".
Then I launch my "berzerker" on HDDs/SDDs/SD cards and whatnots I have on shelves and get rid of thousands of files. It always feel insanely good.
When it comes to old CDs/DVDs (I know with your 17TB these aren't going to be a thing for you but I still have CDs and DVDs from the 90s and 2000s and once in a rare while I'll still burn a DVD), I copy them to my computer, run the berzerker, and burn a new DVD. Then when all is good and well, I physically destroy the old CD/DVD.
I also transcode movies (say family movies) for which I don't care that much about the quality to a much smaller size (say to x265) and then immediately tag the old file as "can always be deleted" in my database.
Also I haven't inventoried every single file. But I'll typically use "jdupes" (much faster than "fdupes") to find the worst offenders: biggest files appearing often are the ones I tend to inventory first.
What's really great with such inventories/tags is that it's something you basically do once. Once a file (and its size + hash) is tagged for deletion, that file has lost the war.
It's over: my berzerker shall regularly run and it shall find it. And the file is gone.
The files tagged for deletion are "dead files walking".
> It's amazing how much digital garbage we can accumulate, I often wonder what my kids will do when I pass, will they try and sift through all of it, or just toss it in a closet somewhere to be discovered in 1,000 years by some digital archeologist?
They'll use a dumb but convenient "AI": find me all the pictures, movies and writings from my dad and sort them, de-dupe them, rate them and show all that to me chronologically and nicely sorted.
At times I'm wondering why I even bother cleaning that mess. I could just wait and have some software sort it for me in 20 years (or if I'm dead by then, my kid shall do it).
Now that said: we're not the first ones to come up with our own DB of files that can always be deleted, always be renamed, etc. I'm kinda surprised there's not yet some open-source software out there doing just that.
Mine is not shippable as it's Clojure code I run directly from the REPL.
P.S: for the hash, to be really quick, I use Blake3 (which may not suit everybody as it's kinda recent but, darn, is Blake3 quick -- and automatically running in parallel).