Harddrive Is Probably Full
marginalia.nu
marginalia.nu
Delete is expensive. "Am I positive I'll never need this again?"
As a consequence, "cheap" actions (saves) accumulate, "expensive" ones (deletes) accrue until cheap becomes expensive as space becomes tight.
Ultimately, options become:
- Find criteria for rapidly assessing what data can be deleted.
- Purchase cheap archival storage and migrate old data to that.
- A media or device crash induces loss of arbitrarily-selected data.
On Linux/macOS you can use `du` from the command line (or ask an AI to make a convenience wrapper for you).
Thats a pretty bold claim, but is it really that much faster?
I think you might he using an old version, or have an issue with your storage.
It took almost 5 minutes and reports almost 2M items, 1.45M files and 465k subdirs...
I don't know that there's anything wrong with my disk, windirstat has always been slow when I ran it on my windows installation. Since it doesn't cheat, it goes slow. I will say, while it was reading, task manager showed a high read rate and a high write rate, and I dunno why, it's Windows (10); not that going through a convoluted filesystem is particularly fast anywhere else.
On a SanDisk portable SSD I own, the cluster size is even 4MiB. If you put a node modules directory in there with thousands of tiny files, it can "take up" several GiB of space.
It's annoying because I'd rather store say family photos than have it full of random junk that means nothing to me. Dunno the answer - better os control? Partition the drive?
https://github.com/Byron/dua-cli
If you're on MacOS and suspect a lot of space is taken by multiple copies of the same files, use fclones to turn them into APFS clones. (You can also use this on Linux, as ext4 supports hard links, but only if you're sure the duplicates won't diverge.)
It found 114GB of Hugging Face models, 17GB worth of models in an obscure app I tested once. In places I would've never thought to look.
I'd never have never found these myself.
Problem solved. For now!
> I'd never have never found these myself.
You are either the most experienced most incompetent software engineer in the world, or you are lying.
I used to be into amateur movie making and editing, and back then I needed many TB of space for the stuff I pulled off my cameras before it got edited down.
Filling an ssd creates the one situation where nand wear out is a problem.
It is a problem to have a full SSD from a performance perspective. Modern SSDs use pSLC (psuedo-single-level-cells) because they can write much faster. When you write to an SSD, you generally write in pSLC mode first, and once the data goes cold, it will get written back to MLC/TLC storage.
Once you are very full, pSLC gets disabled or becomes less effective, affecting write speeds.
There is also an effect known as write amplification that is exacerbated when a disk is nearly full. You want to write 4kb, but the available erasure zones are all full (either valid or invalidated sectors), so now the SSD has to erase a 4MB zone and compact the data there (along with your new 4kb) to write your 4kb.
This gets even worse when you have a filesystem that isn't SSD optimized, such as EXT4. It will cause further write amplication.
Keep your SSD's <80% full if you want good performance.
I bought 16TB disk because I needed that much storage.
If I were using, say, only 10TB, I would be wasting 6TB.
I can expand my LVM as much as I can. But I definitely want my disk to be full. Same goes for my RAM.
And I want my CPU ide time to stay as close as possible to 0% and my load average as close as possible to my no. of CPUs.
Anything else would mean oversizing and overspending.