Ask HN: How do you organize your data and maintain digital hygiene? | Hacker News Reader

Ask HN: How do you organize your data and maintain digital hygiene?

41 pointsKlatchianMist23 comments

The organization of my digital assets is rather ... haphazard! Some files (create, downloaded, received from others) on one computer. Some on another computer. Some files are backed up, some of those backups are not regular. Photos are on the phone, but not on the computer. The phone syncs photos to a large cloud provider, but photos deleted from the phone don't get deleted in the cloud. Videos occasionally get copied to a USB drive connected to the TV. Some data is shared with others via cloud providers, other is not. Files in the various computers are disorganized across folders, filenames, multiple versions in different places.. and so on.

I'm imagining that many of you have clean, well-organized, backed up digital assets across a variety of computers and cloud providers. Curious how you do it in terms of i) Motivation ii) organization schemes (e.g. Johnny Decimal? PARA?) iii) Tools and processes, potentially across file-types (e.g. Photos vs documents, files created by yourself vs obtained elsewhere) iv) Any sharing you do (e.g. with Spouse, family, friends) v) Prevent entropy from growing. vi) Any other points/topics to be considered?

Please share your data organization best practices, warnings, success stories, pitfalls etc.

I created a 'design' for top-level folders and a few levels beneath. A few folder names, jist to give an idea: Work, Home, Photos, Music, Singing, Purchases, Projects, Literature, Ideas, Emails, Expenses, etc. I typically update this folder hierarchy say once every 3-7 years.

I change settings in all programs like browsers to ask me each time where to save. I tend to give long descriptive names to files. I include dates in file/folder names (yyyy-mm-dd - caption) when chronology is expected (like for photo albums).

I typically have an explorer window always open with the currently relevant folders. And use keyboard shortcuts to copy- paste file and folders names when saving/downloading/openimg something.

I use Everything to search when needed. Typically I can reach anytbing just by navigating the folders.

I use big enough drives on all my computers so and sync files between them, typically using FreeFileSync (that handles deletions well).

Phone data is also kept as a subfolder within the above, though the primary copy is on the main PC, except for some data. I use SyncThing and PhomeExplorer to sync files between the phome and this subfolder.

For backups, firstly, the data is already on multiple local machines, uncompressed. I also take full backups using simple file copies (i.e. no compression, so that in case of a failure, the backup is a drop-in replacement. I do not trust 'restore' of a backup program. For making file copies, I use Syncback Freeware.

I really want to do better on this....

But at the moment I use SyncThing, and a 4TB external drive for backups. I use Back in Time, a GUI Rsync frontend.

I organize things generally first by who it's for, then by the project. I have my personal projects folder, and a clients folder.

I don't have any system for anything lower level that that, nor do I want one really. If something is used by multiple projects I'll just copy it and not have to worry about mentally refcounting.

But I might make a folder for a specific type of reusable resource and copy things to and from it.

I share pretty much exclusively via Facebook and email.

I like to very aggressively avoid DIY software in daily life if I can, I don't really have any scripts that are just for personal use although I do have a few FOSS projects, so I don't use any scripts or anything to manage files.

A lot of small stuff like my books and music collection is just straight synced between all my devices. SyncThing is pretty amazing and makes everything a lot easier.

My photos mostly start on a phone, so they are synced as well, if I delete a crappy one on the computer, or move one elsewhere, it will go away on the phone(There is a metadata record left behind by the sync engine though, unless you manually delete the record, I think it includes filenames, not sure if that's an issue for the privacy types).

Code of course is all on GitHub, I don't do any significant programming outside of projects big enough that they have their own Git repo.

I use streaming services and don't pirate anything, so I have less data to manage.

I'm just starting to approach having enough files where 1TB on my laptop isn't enough for everything I've ever made, so I think I will probably move to a NAS or cloud for some things. Backups and not that important files will go on the NAS which will have its own redundancy, more important stuff will be cloud and on my laptop if they fit.

Unfortunately, NASes with RAID are expensive Just the disks cost a ton, and then the NAS itself costs a lot if you want a nice low power commercial box with a remote access service, not some old server from eBay. Maybe I'll try a Pi Zero W 2 and USB disks.

Well.. you asked for it!

I use my laptop running Linux for everything. I store files in the usual suspects (~/Documents, ~/Videos, ~/Pictures) etc. I also have ~/Storage for random files/isos/vm images, etc. I use ~/Workspace for my local code.

I use dejadup to automatically encrypt and backup core files to my remote server that runs my personal site. This backs up files like my ssh keys, gpg keys, environment configs, secrets, gnu pass vault, etc.

My personal server is where all of my git repositories reside. The personal server is backed up every day automatically by Linode.

I have a second nvme drive in my laptop that acts as a mirror of my main hard drive. I manually run a "backup" script every month that syncs my laptop's primary folders to the mirrored hard drive's folders. This mainly serves as a backup in the event something horrible happens to the main drive.

I have an external hard drive I also mirror to once a month. This acts as an external backup, in the event the entire laptop dies (stolen, goes up in flames, etc). This hard drive stays in a fire proof safe.

I symmetrically encrypt my ssh key and send a copy to my wife and a burner email address in the event I need my key to get inside my personal server (assuming my laptop is gone and my external drive is missing).

That's it!

The local files I have to care about are my photo, video, and music libraries, which sit inside Photos and Music on my Mac. Photo/video gets auto synced from my iPhone when I connect it. Code and work documents are on Github and Google Docs respectively, but if they weren't, they'd just have a folder. Photo lib is stored on a secondary SSD because my boot disk isn't that big.

Time Machine backs everything up to a local pair of HDDs. I also periodically create a read-only, encrypted disk image containing my libraries and upload that to Google Drive because I happen to have a lot of free storage there. Before I had any cloud stuff, I used to put one of the backup disks in the bank then later swap it with my other one. My wife also copies some extra special things like wedding photos to her laptop and other disks.

Lastly, some big files I care less about are basically in unorganized places, but not very many. They're either in my downloads folder or on one of the HDDs. I've got another laptop I don't use as much anymore, and my policy is that everything on there is temporary.

For years, I had a system that the structure was mirrored in a synced cloud drive and my notetaking tool at the time, using sort by name to keep its order and using numbers, with the added benefit of just typing the first numbers when in the main folder to point the selection there.

00. inbox, 01. frequent (stuff that doesn't change and I keep coming back to, like an ID copy), 10. ---- (separator), 10. 2023, 11. 2022, 12. Prior, 20. ----, 21. developer, 22. wiki, 23. projects, 24. topics, ......, 30. ----, 31. archives, 32. downloads.

For each year I'd create a monthly folder (as the year progressed), inside a per-topic subfolder with the date prefixed, so results didn't repeat when using spotlight (2022-03-finance, 2022-03-career, 2021-05-retail), then inside the final subfolder with the files related to the event in particular, still with date prefixed.

Topics and Projects inside is a bit like PARA, which I added later, but the month by month one has been a staple. By the end of the month I did a sweep on the inbox folder, and put everything into the month folder with event name or somewhere else.

The thing is that during the pandemic I did a few experiments with this system and since then it's been such a mess, because I keep procrastinating on moving it back to what worked, as some things I tried later seemed to work but didn't. Things like... keep a lot of scattered notes in a doc file per topic? Or separate in my wiki as markdown? Merge 'topics' and 'wiki'? What about the documents related to an event, like a new broker account with all their contracts, signed stuff and such... less subfolders and more verbose filenames? Or more subfolders and more frequent archiving to keep old stuff out of sight?

I usually had this routine of tidying up in the days near the end of each month, kind of mirroring what I did in my day to day job in the office. Ever since commingling different spheres of my life, testing new tools and so on, it led to excessive cruft, duplicates and less organization overall, and I was just looking at improving my system where it fell short.

My warning is, if you devote yourself so wholly to order, the record of your life will be all metadata and no data. I've been obsessed with this topic, as with dust free window sills, and coffee mugs with perfectly aligned handles in my cupboards; there come a day when the diary says 'read and organized diary', and those days are not days.

So I look at my desktop with 5000 files and 40 types, and with sort by date, I can approximate no worse than my own memory, a liner stream of conciousness I occassionally dump to a thumb drive or raid, but no better.

They did survive in the 70s with none of this!

Beautifully said!

I'm moving away from storing all of my stuff on client machines (notebooks etc).

Instead I use an efficient homeserver with Proxmox and encrypted ZFS including zfs-auto-snapshot to automatically manage snapshots, storage is one single 2TB NVMe. This is my "workhorse", where I login remotely and use web interfaces for nextcloud, OnlyOffice and so on.

Every night, I use `zfs send` to backup all snapshots to my Dell T20 width 2x10TB Seagate Exos.

For notebook clients needing to backup files I use restic to store on one single external hdd/ssd per notebook - depending on the use case. From time to time, all this goes to my Dell T20 again.

For backing up Photos and Videos from cellphones I also use my Proxmox Server - an `immich` LXC container with client software on every phone.

All this can be reached online via DynDNS, using OpenVPN port sharing on 443 with an NGINX reverse proxy, so that I can either connect via OpenVPN client or using a WebInterfaces for most of my proxmox self hosted stuff.

Off site backup is also a T20 with my parents, same config as mine syncing via zfs send.

I used to spend a lot of time tweaking this but have had a stable solution for years now.

1. I host seafile on a vps and run seadrive on each machine with asubst to mount it as the T: drive. Everything one needs to keep is saved there; WAF is good. It handles undelete, versioning, etc with access from anywhere for laptops. Weekly my nas rsyncs the data and SQL home for backup. If vps dies I could change DNS locally to point to the NAS, start seafile and continue to use it. A desktop with a seafile client also caches everything locally. I organize with a logically directory structure. The nas does zfs.

2. I also use nextcloud for my contacts, calendar hosted on the VPS. Similar backup home but nightly.

3. FolderSync saves nightly photos of our phones to the NAS at phones\name\yyyymm\ as we don't use cloud storage. Took me some years and a broken phone to realize I need to do this. For convenience, photoview running on the NAS to review them easily. Weekly Borg backups to the VPS for offsite.

4. Something similar with paperless-ngx. I've been meaning to run a personal and work instance instead of one joint one.

I use google drive and google photos.

Photos go in photos, everything else goes in drive.

Within google drive I follow a getting-things-done-ish approach. I have two folders at the root -- one for general filing, one for current projects.

I use a combination of google keep + taking pictures of things to capture things quickly that need later attention.

This isn't perfect. I don't like having to use google docs for all text, for example. Would much prefer to use plain text files in most cases, but they aren't editable within the google drive ui. Luckily they've released the pageless layout relatively recently, which improves the experience a lot.

The pains I have are not worth investing in a custom setup. I just don't have time for that. I want to spend my time accomplishing things, not organizing files, or even worse -- keeping my organizing system online and constantly having to tinker with it. I just can't be bothered.

I'm happy with the fact that at any moment I could have any of my physical devices stolen, ruined, etc. and I would be mostly back on my way as soon as I get another.

Ummm.. hate to play the devil's advocate here, I am sure you must have 2FA setup and all, but in the off chance that you loose access to your google account(for the sake of argument lets say for an "alleged" violation of ToS).

How do you plan to regain access?

I do think sometimes about moving to a google for business account, but the likelihood of losing access to a google account seems pretty low, all things considered. It's easy to forget that here on hacker news, certain probabilities, dangers, etc. are overstated by extremely tiny groups of outspoken people. E.g. based on the comments here you might imagine that most people are somewhat wary of big tech because of privacy concerns, when that's probably not even true of people browsing hacker news in general, let alone the general populace of the US or the world.

I started using syncthing recently and it's just so good.

My approach in the future is a home folder shared across all devices, although my phone only syncs a subset of those folders.

The only deviation from the usual MacOS home directory layout are the Settings folder (for all app configs worth keeping) and the Artefacts folder (for memories that are not photos). There is also a Projects folder that's mostly for code.

I sort my important documents folder by year, since their importance often correlates with the fiscal year.

The Downloads folder is a dump of temporary, disposable files. I keep it somewhat clean.

The result is that I know exactly which directories to back up, and that everything outside of my home directory is effectively disposable. It takes me about a day to get a new laptop in full running order.

The only issue is that when I'm at home, all copies of my data are in the same location. I'll fix it soon, hopefully before it bites me.

Dropbox with important data—not backup, just in case the computer dies on me.

Books and music synced across several computers. Not backed up as I can redownload them, but would prefer not to.

Documents synced with iCloud.

Movies and TV Shows on a small server (a Mac Mini) with a couple of external HDDs.

---

My Downloads folder is my inbox, as anything created by me is already in its proper folder. I go through it at least every day and add each file to the relevant library. Every once in a while, I just delete everything in this folder. I only take photos with my iPhone, which is synced with iCloud. I mostly write notes (Bear), todo lists (Things). Anything else is tied to a project, and I have something similar to PARA for them.

I chuck it into unorganized piles (file system, Google Drive, Apple Notes, and Obsidian Notes) and have good search systems and make sure I know how to use them.

It works fine and doesn't require fiddling or cleanup time. I can find what I need pretty quickly.

I run a zfs array on a home server. The server's off most of the time, so I don't have any fancy services running from it.

When my phone gets low on space, I move my photos and videos across via plain old USB.

Same when I reformat a machine: I gather all the files I want to keep into a single folder, then move it to the server.

I used to run zfs with one redundant drive. I had a drive failure, bought a replacement drive, swapped it in, and had zero loss to my knowledge.

I later read that drives tend to die around the same age, and doing a drive replacement is a very strenuous activity, so you shouldn't just have one redundant drive. So now I have two.

I'm a windows user so I put files I want to store long term on onedrive. If it's a really important file (old photos), I have an external hard drive for backups as well.

Code is all backed up on github

Files that I don't care about losing someday remain local on my PC and those get messy. Most things end up in my downloads folder and I delete them every couple months.

Every couple years I'll get a new laptop and my strategy is to move over nothing. No settings, no files, no wallpapers. Just start over from scratch

It was painful, but I managed to collect all my data onto a single 2TB NVMe drive in my laptop,

folder structure is kinda messy, but under Windows "Everything" gives me instantaneous search, so sorting everything perfectly is not worth the time I run a duplicate finder periodically to avoid wasting too much space

monthly backup via duplicati onto a USB HDD and continuous Backblaze backup

My "organization scheme" (as you call it) took me years to refine and match my brain/personality. Seems to be interesting to others also:

https://github.com/slowernews/hamster-system

Git, ZFS, Raid, 3-2-1 Backup Rule[0], and LOCKSS (Lots of Copies Keep Stuff Safe)

[0] https://www.acronis.com/en-us/blog/posts/backup-rule/

I am also disorganised but there are attractors of important data:

Bitwarden: passwords if course but also acts as an index of all online accounts and other private info.

Photos: one folder on one computer with iDrive backup

Email: lots of important info there. i need to set up some automated backup though

Microsoft Todo, Google calendar.

Chucked it all Dropbox for a decade until they increase the price by a lot (iirc?) then I moved to iCloud Drive.

Reply on news.ycombinator.com