Digital clutter: Learning to let go and stop hoarding terabytes

NO!

I will keep adding an /archive folder to every PC I own and copy the complete contents of my previous /home/ folder into it, including an endless amount of recursive /archive folders.

I will never look at any of those again. But Archeologists in the far future will find my data and it will revolutionize their understanding of our time.

I recently went through 35 years worth of cassette tapes to pull off old songs I had written as a teenager and songs I played with bands I was in over the years. I’m so glad I saved it all to digitize. (And glad I finally got around to it.) I found one song of a band practice from before our band had a drummer or bass player. It was just me on keys, the guitarist and the singer. The guitarist is now dead. We never performed that song live anywhere as we ended up cutting it from our set because it was just a little outside of our style. I was able to bring it into Logic, clean up the tape noise and add a drum track, just to see what it would have sounded like. It let’s me relive a moment of my life in a way that just a simple memory couldn’t. I also found recordings of shows we played. So glad I didn’t throw any of it out. (And yes, some of it was terrible. I didn’t digitize those parts.)

> I ended up deciding to get a small, simple external Thunderbolt SSD enclosure.

See this is the problem with the whole minimalism/decluttering/etc. movements: half the time they are so entrenched in consumerism that they authentically believe that buying this new thing will solve their clutter problem. The author goes on to buy a new cloud storage service that is "a backup power tool for advanced users". And spends all this time doing all this migrating and reorganizing activity that seems an awful lot like work.

People in this trap will talk about how minimalism and decluttering are about making their lives easier and simpler, but that's not what I'm seeing happening. This seems harder and more complicated. I don't want to have to be an "advanced user".

I fell for this for a while myself, but no longer. I want to be a dumb user. There are other things I want, like security and privacy, and control of my own data, and to his credit the author does mention that, but if that's your goal be focused on that, not this not-quite-sensible form of decluttering I keep seeing popping up.

I work with photos and videos and cannot comprehend downsizing to 2TB. Or even finding time to go through and selectively cull old footage. I shot a simple one-day drone-only vineyard job last week that generated 140GB, not including graded renders and photo edits afterwards.

On any given day, I will have 5-7 external drives connected, plus a NAS box. There are another 15+ drives (2-5TB) in the drawer beside me. SSDs as drives I actively work from and platter drives for backups or rendered footage.

I feel like the secret is to nailing the workflow at ingest/render because it’s painful trying to going through en masse and a year later.

Edit: I’ll add that I think one problem is that in shooting with a drone, there’s less to cull. Everything is in focus. A high percentage of the photos are usable in media libraries for the client and about 95% of all video shot is. My wife is a photographer and far fewer shots make the cut because of focus, or a facial expression, etc.

I know this is talking about large files like photos and videos.

And I get the whole storage is cheap line of thinking.

But there’s actually a different problem outside of photos, etc. Simply, index pollution.

I have lots of little projects on my machine. I have lots of open source code on my machine. And, while I can’t think of any specific examples, there are areas of search on that, dare I accidentally trip into that hole, are just filled with detritus and garbage.

My Spotlight is gorged with false positives for some terms. And I know that I have searched for things that I know I have, but unable to locate, or at least certainly not easily, because I was searching “wrong”.

Indexing provides lightning fast access to everything I don’t want to see.

Mind not much I plan to do about it. I guess I can take some “never again“ stuff, put them on an external drive, and tell Spotlight to ignore it.

That said also, my phone is full. About 2g free. Mostly photos and videos. Mostly my cats. Solution is simple. Need a bigger phone.

Not sure how are these nvme enclosures supposed to work. The problem is right in the picture. Samsung SSD bla bla 3.3V/3.3A = 11W and that's just for the SSD itself, the controller chip in the enclosure will also consume a lot of power at the advertised speeds. So you're looking at say 13W or so peak power consumption of the whole deal.

Normal USB 3.0 port can do 4.5W. So sticking that contraption into such port will not work reliably at all. Specially marked USB 3.0 port can deliver maybe 7.5W. Type-C ports, who knows. Depends on what the port advertises via pullup-resistors. It may be 4.5W, 7.5W or 15W. You never know.

Nice in theory, but none of the usb nvme adapters that I bought work reliably in any port for longterm use.

They certainly don't work in any low power devices, like various ARM SBC USB ports because most of those use current limiting power switches for USB ports.

And they don't work in my workstation reliably longterm either, for whatever reason.

I guess you really have to be lucky to have a 15W Type-C data port. Funny how often these sell with Type-C <-> USB-A cables, though.

Regarding Arq backup: if you are worried about using a proprietary (enrypted) and closed-source backup format in case the company were to go under, they have an open source command-line restore tool:

https://github.com/arqbackup/arq_restore

I've been using Arq for years, but I need to look into the "Glacier Deep Archive" format which is about 1/20th the cost of the fastest storage class.

I work with a vast collection of unprocessed photographs, which takes up about 6 terabytes of storage space on my 4-bay network-attached storage (NAS) device - a DS418. Despite being limited to 1 GbE speeds, the process of loading each of the raw images, which are around 60-70 MB in size, only takes half a second. I occasionally use the smb multichannel flag to improve my bandwidth when connecting to my Mac through two gigabit USB-C Ethernet adapters, but it doesn't make much of a difference. Editing over Wi-Fi using my Wi-Fi 6 connection, which usually runs at 700 Mbit/s, is also not an issue. Although having a 10 GbE connection would be nice, it is not a priority since the NAS hard drives' speed would become the limiting factor before reaching that level, and the cost of upgrading the hardware to support it is prohibitively expensive. Adding a solid-state drive (SSD) cache to my NAS could be beneficial, particularly when using a 10 GbE connection. Synology provides 10 GbE add-on cards for a small portion of their stations.

Storage is cheap, local networks are fast, ZFS is great, and NFS just works. I'll keep hoarding, thanks.

" I had gigabit ethernet but that doesn’t matter, these spinning disks are barely faster than 100MB/s for sequential read even with a RAID setup."

Umm doesn't gigabit ethernet basically give you a max of 125MB/s anyways? And I was seeing transfer rates of sequential read from 7200rpm SATA drives greater than that even 8 years ago...

If you choose SSDs, remember to power them up every few months or so to keep the data alive.

In my 20s I got rid of some stuff on purpose, and got rid of some on accident. It's a real loss. I LOVE to time travel though all my old files and it's a shame I don't have the first years of programs I wrote, a bunch of emails, all poker hands, etc.

I'm still looking for my wallet.dat in any of numerous old backups hoping it still has some early Bitcoin I mined when it was still novel and Bitcoin faucets were a freely accessible thing.

What about “tree shaking” the stuff you don’t care anymore?

It’s not easy to know what to delete on the present day but some years later it is. I never regreted having deleted anything!

What about deleting the worse photos from that hike 10 years ago? If you don’t like them now, most likely you never will… and nobody wants to rewatch hundreds of photos from that hike. Not even you.

Nah, too many files have disappeared completely from the Internet, no one's interested in archiving and no, archive.org doesn't get everything.

I'm the opposite. I am selective, but it's work to decide what's worth keeping, especially if you are prolific with your camera. However, two years later I have more emotional distance and have grown as a photographer, and it's easier to discard swaths. Bulk storage plus time makes easier.

It's the same approach I take with tax documents. Roughly group by year, yes, but why carefully select which ones I have to keep now, when I can just throw all of them out in a few more years?

While it's admirable to hoard everything, and future historians (read: most likely inquisitive relatives) might be lucky enough to inherit one of your archives that hasn't succumbed to bit rot (which, even with proper storage, may be well under 10 years), don't overlook the relatively simple way to maximize the chances of your images being enjoyed by people in the future and today: make prints!

The best way I've found is to make photo books. Most companies use print technology that lasts upwards of 200 years https://your-digital-life.com/long-will-photo-books-last/. Print a few books, give them to a few relatives, and you can be assured that the best of your photos will be viewed for decades to come. This way you can share your best while still hoarding that archive of every. last. photo.

Thank you for reminding me that I need to duplicate my 15 TB storage, just in case.

Every time I cleaned up I regretted it a few weeks later so I stopped doing that. Storage is cheap. Keeping storage alive is not cheap however so just power down the drives and maybe spin them up once every 6 months or so, so the bearings don't seize if it's spinning rust.

Bad take alert. I have just over thirty years of my own photos (scans too) and a lifetime of my parents photos to go through.

Time has the quality of making the ordinary extraordinary and the race to purge one thing to make way for another can be passed on as a task to the next generation.

I bias towards keep rather than discard both because cost is low and because the tools are getting better. About 20 years ago I even realized I don't need to discard the blurry pictures because someday they might be recoverable or otherwise useful (that day is probably here).

And as personal (local) search gets better with smarter systems it may be that future codebases will surface interesting insights or memories of some long ago event or activity as it can do with photos today.

However when my parents pass on I will discard all the landscape photos and and all the photos of long dead relatives I only met as a kid and don't even remember. Some of those are quite meaningful to my mum and dad, but are meaningless to me.

I still have some .mod, . S3m, .xm music files that predate mp3s :)

Currently sitting on 250gb of RAWs since October, at some point that will hopefully become 40gb or less. The last time this happened I batch converted everything to Adobe DNG, but support for that isn't great outside Lightroom. Is there some format with good compatibility somewhere between DNG/RAW and JPG that preserves dynamic range and white balance information? I think exposure and WB correction is pretty much the only functionality worth fighting to preserve. Keeping RAWs for long term personal archival seems silly

Hoarding terabytes is sort of a thing you can never get your head around unless you are stealing mp3s.

But hoarding gigabytes is pretty easy, also I think the term hoarding is pretty loaded language in this context? Saving things so they don't get lost is the way to go.

Just as an example, after Steve Jobs died apple went into everyone's email at .me and deleted their emails between them and SJ.

You probably didnt see that coming.

> It's when you start looking for 4TB SSDs that the prices go up considerably

That’s not my experience, having recently built a PC for myself. SN850x 4TB was about 2x the 2 TB price. (Depending on the day, it was -15% to +20% from the linear price, but was usually in the 3-5% higher than linear.)

I didn’t see a reason to go small for a couple hundred bucks of delayed purchase.

I asked chat GPT-4 to write me a rust program that recurses all files in my external drives, generates a xxhash of each file, then inserts it into an RDF graph.

That way I can scan for any files I don't have centralized before wiping out each older drive, and query for specific cases.

It's also nice to recurse the old unimportant video files and re-encode them with ffmpeg vp9 or av1, opus, so they don't take up as much space. Get rid of those raw video recordings and xvid/divx codec videos.

Re-encode all your old 7z, .zip, bz2, .tar.gz into zstd compressed files. It can also save a ton of space.

Buy manufacturer recertified HDDs, you can get like 18TB for $190 or so. Buy a few and put them in a truenas server. Hoard all the data you want, forever.

The "Article" is just an ad, disguised as blog article, for affiliate links. I don't mind actual mental health discussion, but this is just bullshit.

I took up bird photography about 3 years ago. I have 400,000+ RAW photos in Lightroom on over a dozen 2 TB SSDs. It's not so much hoarding as it just takes effort to go through and decide what to keep and what to delete. I shoot bursts so it adds up really fast. I could probably get by with 10-20% of the storage if I could just keep up with the pruning.

I did eventually throw out my floppy discs and a couple of years ago I finally scrapped my windows 98 PC after transferring files off the hard drive.

I still have data going back to around 95 though, and I like that. Pictures and chat logs from way back when I first met my wife. Old games I played as a teen. Some of the code I wrote way back when I was first learning and some of my first open source contributions.

I did lose a lot along the way, though. Sometimes I wish I could still see some of that earliest stuff. And things that have been lost on the ephemeral 'net - old BBS discussions, usenet topics, my teenage livejournal, etc. that I didn't copy and save.

The tough part is that it gets hard to manage.

While I do pull forward the most important stuff each time I get a new PC, a lot of the rest is sitting in old backup formats from various backup software that I might not even be able to restore anymore. It's always nice when I think of something I used to have and can go dig it out of one of those backups, but I don't know that I will always be able to.

And it can be really hard to find something. I've tried various organizational schemes over the years, but that just means that some things are organized one way and some are organized another, and it's even more difficult to find things. I suppose I should go through and re-organize everything into one consistent standard structure, but that's what I've always done before and it just becomes like the XKCD about adding yet another standard.

Anyway, the digital clutter still works well enough for me. It's sort of a cozy old home filled with sentimental things instead of a sterile empty monastic cell.

"I just wish Google Photos wasn't owned by Google. I'm trying to reduce my reliance on consumer-oriented Big Tech products" after discussing Amazon's S3 & Glacier products seems odd.

I recently nuked about 10TB of data and decommissioned my Unraid server. I do have a disk on a shelf with the contents of that data If I do need something from it I can dig it out, but it is there to rot.

This blog post shows S3 Glacier Deep Archive costs $2.05 USD for 2 TB but on the AWS pricing page it's showing as 2,000 GB * $0.0018 per GB = $3.60 ?

Edit: this is for London pricing, guess it's a bit higher.

If terabytes seem like a lot today, will they in 10 years?

250 min hd was a lot once

Then the 1 GB hds.

Storage will go in lockstep with video quality online and on device.

What’s a reasonable amount of storage for 8k video?

Is Arq open source? I have a similar problem set but I don’t want to use a closed source product managing my encryption and uploads.

The submission title promises a philosophical discussion but the contents is just about technical stuff... Disappointing.

You'll own nothing and be happy. c:

Embrace the sublime minimalism of leasing your data from someone else's computer.

Am I the only one who hates having to keep anything? As a kid I would go out of my way to not be in photos, and as an adult it's furniture and appliances.

If it's important it's probably somewhere in my email, but wanting to deliberately store personal data locally or in the cloud makes my eyes roll. Who cares?

welcome to the club