Backups Aren't Simple
filipovski.net
filipovski.net
The first was when the telephone pole outside our house was struck directly by lightning. Not only was it the loudest thing I have ever heard, the current surged through the telephone line, into the internal fax modem, and fries everything within its vicinity. I was 10. I did have backuos, but only only floppy and they didn't cover everything.
The second was storing data in OneDrive - a change to their terms surrounding "lifetime" unlikely noted storage, combined with a client that was unusably slow to download and a deadline for data retrieval meant that I lost most of my files.
The third was SD card failure in digital camera on holiday, the controller chip died catastrophically, leaving the card completely unrecognised. It was a brand new Sony 128GB card, manufactured by Toshiba, and it seemed to be a common issue. I now shoot to two cards simultaneously.
And the fourth time was ... Performing a backup. An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup. I've been weary of using rewritable media for some time now as a consequence, but I think backups themselves are high risk activities.
A very lovely change over the last 8 years or so lol. I came up in film during the DSLR revolution. 5D2’s/7D’s/Rebels (i series) years.
8bit 420 nasty aliasing recording on single cards and praying baby. Magic lantern booted on those same SD’s!
GPS is useful, but not wifi. I don't need my camera becoming another IOT always online telemetry device. They're one of the few electronic devices built to last. Imagine needing fingerprint unlock on a camera, and for what? Prevent others from viewing your photos? Changing your settings? Also copy from card to pc is the best possible workflow. Imagine cameras like smartphone with no expandable storage.
Stop giving these people ideas! I am now very afraid they will listen to this guy since he actually has a follower base.
So turn off the camera, open the memory card slot cover which is flimsy 1mm prong, eject the card, insert it in the reader (if you have SD reader) or fiddle the microSD first from the convertor and then insert, copy, (insert microSD to SD convertor), open up slot cover, insert the card.
Surely beats just connecting the camera to the WiFi (which it does have anyway) and downloading through SMB. Suuure.
S9900 from 15 years ago had a built-in GPS receiver (and absolutely unneeded and never updated POI system). P1100 needs a smartphone for GPS for whatever reason - and app doesn't work 5 times from 5. But the latter is on me, I knew what I would get when I bought Nikon.
pro camera manufacturers are not google or apple. Have we became that defeatist?
just look at the chinese import ban on both mobile phones and cars!
Btw X100F has Wi-Fi and automated file transfer to PC. But it is faster to transfer it via SD card.
And the rest of it is essentially features for a market that doesn’t exist. People buying cameras today don’t want to send straight to Facebook. They want to sort through raws, edit in Lightroom and then post from a laptop. The market for the features you want all moves to phones long ago.
Sony also has a bunch of the features you are asking for. It can connect to the internet and live sync files to an ftp server. It can even upload proxy videos so your editors can start work before you get back to copy the master copy over. Cameras are sold to prosumers and actual professionals now so the featureset reflects it.
Pretty sure that's the main point of Fuji camera, the output and built it filter is decent enough that you can upload to Social network without any editing.
So why can't cameras do it? They are the same form factor, could be a feature to enable. My smart garmin instinct can track gps for a day, and it's a tiny sportswatch.
Sure you could put a mobile modem and internet connectivity on a camera, but it adds price, reduces battery life, and doesn’t really have any advantages over a phone companion app
Wifi NAN could be used to solve this problem but I’ve not seen anything using it. Apple only just added support last year though.
The 5D3 was first, so that was like 12 or 13 years ago now. It rolled out after that over a few years across their fleet.
I had a similar experience with a Chinese cloud storage provider. They didn't even give a convenient way to export the data. And the client throttled download to like 100KB/s. I luckily was able to parallel download by running the client on several VMs...
> An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup.
Sounds like the backup workflow is fundamentally flawed... But I also have the fear that the backup program with root privileges can go off the rails... So I keep my backup job as simple as just running restic with some systemd restrictions.
I briefly considered using bluray disks as a backup for my photos and other critical docs. But getting a decent bluray burner seems not so easy these days with most production winding down. Next best thing looks like the "object lock" feature on object store services that prevents deleting objects for a certain time.
In terms of preventing "oops" moments, I'm mainly relying on software (restic) for that, where I trust that (A) backups always append data rather than replacing and (B) it's logic works for marking which data to purge based on rules is accurate. [0]
[0] https://restic.readthedocs.io/en/stable/060_forget.html#remo...
While the local backup service uses an API-key with limited privileges, I need to go back to the bucket/storage-zone settings and see what I can enable, how it would affect total storage, and whether I (with additional privileges) can reliably grab the correct mix of old-blobs to do a restore.
On the other hand, LTO-5 drives are now pretty affordable. And each tape cartridge holds around 2Tb of data for about $20.
I had a smaller setup before, with a simple external LTO-5 drive. Used drives are now are about $300, and you can probably find them cheaper. And LTO-5 is the minimum realistic version, it's the first one that supports LTFS and it has reasonable tape capacity.
Lol, no. "As of today we are closed. Goodbye."
Especially considering what giving even a two weeks now considered "generous".
The probability that the object store goes out of business at the exact time your own copy dies is insignificant.
Or even fancier - just changing the price for the egress.
The probability of a hard drive failure or ransomware at the same time as backblaze or aws going out of business is pretty much not worth thinking about.
The probability of an attacker using the api key in your backup script to destroy the backups is far more possible.
online backup should not be your primary method of backup, you're one billing, identity theft, financial issue, or health issue away from getting your backup nuked
do you have PBS clout? if not good luck https://arstechnica.com/information-technology/2026/08/pbs-s...
It shouldn’t be the only copy of the data that exists.
It was such a tremendously silly decision I almost wonder if it wasn't a politically motivated decision. Maybe the provider that went under had some relative DEI points versus Backblaze or AWS. Just bizarre.
But I wouldn't extrapolate anything out from their situation.
So much fail in one single product.
Either give me unlimited space or get out of my file system. They could have just made it a separate thing and let me put stuff in there if I wanted to, instead of hijacking my file system and just messing it up.
I stopped running Windows wherever I can not because the tools or OS are that bad but because a machine should come with sane defaults but should respect my wishes if I want to go against those defaults. When it starts to override me, it's not my machine anymore and whoever "owns" it (MS in this case) can shove it.
I have a Mac, so not just an iPhone, so I use Arq there, it materializes the iCloud stuff as necessary for backup. But if all you have is an iPhone or iPad along with an external disk, NAS or whatever, Parachute Backup will do backups to a variety of destinations.
It's mainly just rsync. When the phone was plugged in, the photos were accessible at a path like "/run/user/1000/gvfs/gphoto2:host=Apple_Inc._iPhone_abc123/".
Now I use Time Machine, but I still need to check whether it includes iCloud photos.
The script currently requires changing one or two folder paths at the top (more details in the script).
So I end up with 3 copies of all my photos, on 3 different providers (I mean... 2 different providers plus my homelab disk).
[1] https://immich.app/ [2] https://github.com/garethgeorge/backrest
The answer is : it's hard. If you only have a iphone and more photos on your icloud than can fit on your phone, you're kinda screwed if you don't know what you're doing.
AFAIK Apple makes it very difficult to get all your icloud photos bulk downloaded, original quality off icloud.
Google is no better.
It's much easier to start backing up from the beginning but once you're stuck on icloud it's hard.
If you have a Mac with sufficient storage, you can just enable iCloud to store photos locally, then export them. That could miss some data (I think descriptions you set for a photo were not exported the last time I checked), but it is most of the data.
And, I may have conflated issues: Iphone Storage < Icloud storage vs opposite way.
I would imagine this doesn't work well after a certain amount of data. I’m also not sure they were the originals.
I guess it's because the core offering is still some version of "a folder that syncs", and the photo gallery is just a view bolted on top.
I steer clear of any solutions scoped to photos only, as it always turns out they either screw you up by reducing quality and throwing away the originals, or trying to lock you in by only offering metadata they keep in a database they will not let you export.
I remember seeing a longer blogpost re the topic of retroengineering the Apple phot sync story, but I could not find it.
[1] github.com/rcarmo/PhotosExport [2] https://github.com/craigtrim/icloud-photo-export [3] https://icloud-photos-downloader.github.io/icloud_photos_dow...
This made me remember the time during the 90s when I lost all my BASIC programs due to a hard disk crash, and seeing me sitting sad, my mother handed me a bunch of floppies and asked me to see if they contain something useful.
And turned out that it had all the programs! I took the backups and forgot about that. Apparently that is one of the hard things with backups. You need to track them...
startup opportunity.
oops ...
What kind of camera put shoots to two cards simultaneously?
Edit: TIL about cameras equipped with dual memory card slots
We are not in the backup business. We are in the restoration
business.
"You know how to take the backup, you just don't know how to restore the backup. And that's really the most important part of the backup, the restoring. Anybody can just copy a file!"
Before I could finish the greeting I knew it was going to be a wild call as I could hear in the background someone shouting and someone sobbing, the guy actually on the phone sounded like he wasn't planning on making it home that night.
Turned out they were testing out some sync software and after a test they deleted the sync without turning off the app which then did its thing and now nobody could log in except admin who was the only user left.
I was able to find and confirm that the backups were in place and accessible. Which was good, I could hear them breathing again. But I had no way to get the restore going because the source server was so hosed.
Just then the dev walking by and I got him on the call and he did some outright hacker magic to get things to the point that a restore could function.
I thought about writing my notes into a troubleshooting guide but then realized it would be better to skip the hacking and make a setup guide that showed them how to set it up properly in the first place.
No. But, well, yes: if you use BE - you will need a restore pretty soon.
I'm the person who actually experienced BE walking around the file server and deleting files. Helpful suuport person had done the needful and said what this is not an issue and it wouldn't be reported as a bug.
When I was a teenager, I was the reason for data loss for my dad, twice. Both times it was because I was re-partitioning a hard drive to install linux.
You would think that taught me a lesson about backups, instead it reminds me every now and again to be grateful for an awesome dad and aspire to handle situations with my kid similarly :)
Also "There are just two types of equipment; those which are already broken and those which will be broken".
I've found a good way to point out anyone thinking right direction not to trust their valuable data in any single entity, product, location, etc.
When I heard it decades ago IIRC 80's Nokia Data Unix courses I went. The distinction made Backups is just 1/3 of the triad Redundancy, Backups, Archives.
1) Redundancy is what you get using RAID, multiple network connectivity etc. ie. good for avoiding single point of failures and avoiding loss service or product availability.
2) Backups are for recovering lost data needed returning lost running known state right before fault or some time state before what backup schedule and used rotation cycle can provide.
3) Archives are meant to saving valuable data not expected to change or active use and but there is need saving much longer periods.
Not seeing and understanding usefulness of difference between these seem hard some people I met over decades. Which then led to confusion, not knowing what they are up to with their data and also often later disappointment when they found out that for example backup programs are not great if you expected archiving instead.
Moments later their dad (presumably) comes back and asks for his phone back and sits down with their food. The kids are eating and the dad is fumbling with his phone with odd looks and I can see his frustration growing.
Then out I hear something like "What did you do to my phone?" repeated loudly over and over. "You erased my phone, it's gone."
The small children (guessing) had figured out how to factory reset his phone.
My thought at the time was something like : How many times did I do something as a toddler I don't remember to piss of my dad like that and that he never got mad or brought it up later (Lots of times knowing myself).
The wise man sees that when a fool has broken his system, its no use getting angry at the fool. Make the system better.
One day I found an interesting new command called DriveSpace, and it looked exciting. I ran it on the C: drive and then got cold feet and cancelled the operation half way through. Needless to say this was a disk compression application, and I had just unknowingly compressed half my hard disk. Took a long time to figure that one out and I was in trouble with my Dad for bit, but we called a few experts we knew for advice and he included me along the journey to fix it. Something I always try to remember when my kids do something similar.
[1]: https://github.com/jimsalterjrs/sanoid
[2]: https://du.nkel.dev/blog/2026-05-16_rootless_docker_virtiofs_proxmox/I have an old and loud 16-bay server that boots every 7 days or so if no one is home, `syncoid there here`, and shuts off. I'll get pinged by uptimerobot once in a while if it's overdue and I'll get a notification if a pool is unhealthy or reaching capacity (`sanoid --health` I think). Otherwise I forget I even have it set up.
Automated cold backups are great peace-of-mind.
All my hosts run the same CoreOS setup (https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup.
I plan to implement it like this:
vps1:
- restic container with custom sh entrypoint that will backup volumes/ to homelab every 24 hours
homelab:
- backrest container, to back up volumes/, do prune/check, replicate repo to offsite
- rest-server container, will store backups from vps1, homelab, offsite
offsite:
- restic container, backs up volumes/ to homelab every 24 hours
- rest-server container, store copy of backups from homelab
The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost
Your secrets.yaml makes me nervous though - too easy to miss a key and leave something exposed. Why not just add the whole file to the vault?
Previously, I was using artix and had this “etc” directory[1] checked in to keep track of system configuration, but there was no good way to keep track of config drift (other than remembering to update this dir).
Haven’t gone through a CoreOS update yet (been using for ~2 months), but doubtful it would break anything. I’ve tested to make sure all my containers shutdown gracefully etc.
The SOPS (secrets.yaml) pattern is more common in NixOS configs, and I found it works nicely here too. In the artix setup, I had a bunch of .example files strewn around [2], which I had to remember to sync with the real versions.
Encrypting a key is just prefixing it with “enc_priv_”, SOPS will encrypt and decrypt it automatically. I keep “public” values plaintext to maybe help someone setting this up for themselves. Just have to double-check git diff before committing.
[1] https://github.com/ebrahim37/infra-template/tree/00eccff06ae... [2] https://github.com/ebrahim37/infra-template/blob/00eccff06ae...
It is a tool that was created by the author of sqlite. It does just what you would expected to do.
"So what is the WiFI password here?"
"You sure would to like to know that eh you effing hipster"
"Does that start with a capital Y?"
"Yes. No spaces."
This is an empirical claim, but is it grounded in reality?
99% of people don't backup, and that's probably the right choice because the risk is low, they can't meaningfully improve their restoration rate themselves and they don't care enough about their data to classify its loss as catastrophic.
When a person hears someone talk about the importance of backups, but also haven't heard friends/family suffer this date, they will rightfully ignore this warning.
Or do we all follow the best practices as it relates to backups, exercise, sleep, nutrition, accounting, house maintenance, ... Etc.
I do not. Backups aren't near the top of that list.
This stuff just isn't very important to most people and that's okay.
This depends on your viewpoint. Using gmail for example is an implicit backup of your e-mails. Uploading your pictures to Facebook the same. The move from local, private compute and storage to the cloud solved this issue for many to a degree that it is now less important.
Over 40 years (even with changing disks, upgrading PCs, etc.), the chances of you ending up with your data intact is 0.99^40=0.67. May not seem too bad, but this is just a hardware failure of your main drive we're talking about. When you factor in all the other types of screwups that happen, the annualized failure rate will skyrocket.
So a more conservative version of this quote is that a 20-year old today is very likely to experience data loss over his lifetime.
Thankfully with cloud being ubiquitous at least photos are given one copy in the cloud, but IMO even that is risky. People have different appetites for risk, and that's OK. But over the last 20 years the all of our data has become pretty important. I don't know how many people could stomach the loss of important documents, photos, etc.
It’s still vitally important if you are self hosting your data.
Maybe they just haven't talked to you about it? Most people I know have suffered some form of (partial) data loss or another, and lived with it.
* Consistency groups. The article kinda almost gets there, but stops right before it realizes there's a problem. If you have multiple programs accessing the storage, s.a. a database (that you are snapshotting) and the application using the database, then you are risking an inconsistent snapshot when the database state you wrote is ahead or behind the application state. You need a mechanism to freeze all I/O beside the snapshot in order to ensure that the snapshot data is useful.
* ACLs. And similar external mapping to the storage being used, eg. what if the users come from an LDAP server? Similar problem with symlinks coming from/going to ouitside the snapshot.
* Sparse file support. Let's say you have a few VM images in your snapshot: if your tool doesn't support sparse files, it will likely greatly inflate the image sizes.
Bottom line, either use the product's own snapshot functionality (ZFS has those), or switch to the product that does.
One of my teachers claimed that contrary to common behavior, no one actually valued backups; it was restores that were worth paying for.
On the flip side of that though, really and truly, I've restored a laptop from a timemachine backup and it came back damn nearly identical and that was magical.
I needed to temporarily move off 20TB from my NAS and bought a Hetzner storage box.
I wanted to encrypt the data before sending it so i went with the borg ssh mount. Long story short, borg failed a couple of times due to https://github.com/borgbackup/borg/issues/7672, i resumed it per documentation and got 2gb of silently corrupted data. There was no way to do data integrity checks due to broken pipes and there was no option to resume the job from the previous point either. I couldn't afford to go with the s3, backblaze wasn't an option due to placing trust into their proprietary client doing the encryption and i got majorly screwed in the end.
Also you must have been unaware of Backblaze B2? It doesn't have a proprietary client, and it launched 11 years ago only a couple months after attic was forked to create borg.
We charge our customers for backup storage by volume and that includes an annual demonstration of recovery. We fire up some or all of their systems on our gear in isolation and show that they are reasonably functional.
Your teacher's comment is riffing on the well trod lines of: "heights don't kill people, it's depths that kill people", which is all about perspective. This is is not related to "guns don't kill people ..." which is about agency.
When data is lost, or decay takes over, the value can skyrocket. Depending on what is lost, someone may be willing to pay 100x when they would have spent on backup in order to get their data back, this is why data recovery services can basically charge whatever they want.
My assumption is the Venn diagram between people who pass the marshmallow test, and those who proactively backup their systems, has a lot of overlap.
Two weeks later they had a fire in one of their huge printing machines which caused quite a lot of direct damage due to heat and rendered a lot of equipment broken due to smoke/fumes. It was also, as it turns out, a week before the first COVID 19 lockdown in the UK.
Their backups landed on a XFS file system with reflinks enabled. I cloned their repo and fired up NFS with their VMs running on our gear. It took me another hour to spin up another OpenVPN server (CA etc) for them to use and sort out a few other details (coffee doesn't brew itself).
They ran via VPN out of our data centre for most of the pandemic.
That was an utter triumph but I also have some rather less triumphant stories about backups and lack thereof. Let's skip over those 8)
Obviously for production systems automation & monitoring is the way. I kind of like this setup on my own machine though.
https://www.jwz.org/doc/backups.html
(maybe copy/paste this into browser instead of following link, since referrer from hn apparently does something)
This leaves you needlessly exposed to this failure mode: Commit some "braino" that wipes out some recent work. Go to bed without noticing the loss. 5am rsync run wipes your backup as well.
The fix is to back up to not just one but to a rotation of images. Use --link-dest to reduce storage overhead to the size of the directory hierarchy only.
https://relax-and-recover.org/
This is two-fold useful, first as it enables boot from backup media for total recovery, second in that it creates a backup.tar.gz that holds everything that was not explitictly ignored in the /etc/rear/local.conf file.
I'm running Oracle databases on these systems (and the local.conf is configured to ignore the datafile directories). I have standby databases (not Dataguard in that I don't have online redolog replication) that allow me to recover them to the primary in a disaster, with some loss of committed transactions.
I finally have rsync configured for some scratch temporary files.
I have dallied with btrfs snapshot replication for home directories (in a loopback mount). My vendor support (via a CSI number) has been on and off, so I don't use btrfs in areas that we need it most.
My replacements do not like the complexity (and I am retiring).
GNU tar has its own incremental index via `--listed-incremental=FILE`. Unlike Borg or Restic, which have their own more complicated repository formats, this leaves me with just one additional file (the `.snar`) alongside a dumb, portable full-disk tar archive.
The nice part is that, unlike repositories that require both read/write access patterns, tar can compute deltas using only the small `.snar` file, while the main `full-disk.tar` can remain buried in write-only Glacier storage. This makes it a much better fit for Glacier's write-once model and 180-day minimum retention.
My current plan is to upload a full-disk tar to an external HDD + Glacier on a weekly cadence, and the `.snar`-based deltas daily. That gives me a pretty simple cloud backup solution for a few bucks a month (after burning through $100+ of free credits).
For me, the ability to mount a backup and look at a particular file proved to be important. Also, I restored from my backups four times; two of them was moving between machines, pretty quickly.
Software like https://restic.net/ does a good job. Few choices of backend
Accepts stuff from pipe too so you can just pipe mysqldump or pg_dumpall without intermediate files
Very decent options for checking repo integrity, personally (well, at work too, we use it on few hundred user machines and servers) I also added "roll a dice for backup and try to restore it" test script to make sure it is working
Decent deduplication too so "store year's worth of weekly snapshot" is very sensible strategy once you exclude the "constantly changing and useless" (caches etc.) files out of it
can mount FUSE directory with all backups on sensible OSes.
https://kopia.io/ does the same +GUI but a bit worse on CLI front (it insists on keeping local config which makes scripting a bit more involved, but not much)
IMHO, the easiest way to perform a restore test is to use your production backups to refresh a lesser environment on a weekly cadence. Naturally this has to be appropriate to the type of data you're restoring, e.g., our E2E (end to end) testing environment has been built to the same risk tolerances of production so it's suitable for production data and the PII (Personally Identifiable Information) it hosts. If this isn't in your own risk tolerances, you can still perform the same test but ensure the data from prod is anonymised or alternatively destroyed, and perform a second restore after production restore with the dataset that preceeded it that has no PII data or similar.
With this the E2E tests confirm the backups are expected and you can tick a box that when shit does eventually hit the fan, data corruption isn't a problem. Normal caveats apply, you must make sure your backups are immutable so they cannot be modified or changed _after_ the tests have been ran.
We also use S3 versioning in case something gets the keys and just erases it. Then it's replicated to the cloud S3-compatible store to get the offsite location.
Only feature I really miss is "turn this restore into .tar", both for testing restores and for convenience when we get requests of "we need this directory from 4 weeks ago"
Kopia does not let you pipe into it without intermediate files, or at least not in a friendly way I could figure out.
Kopia seems quite designed around the backup server being the place where the config is set. IMO it's harder to configure if you have your own declarative config tooling (like Ansible, NixOS, etc)
The nicest way I found to ignore files declaratively is to configure a ignore-file pattern in the repository policy and then declaratively create `.kopiaignore` files.
The Kopia server is convenient if you have a server at home and want to back up laptops to it, with them being in a shared deduplicated repository without having to trust the clients not to get pwned and wipe out the backups. However I'm not aware of a nice way to then transfer those backups to aggregate them into a bigger repository. (So I just rsync them off-site and don't worry about getting deduplication with other backups) Also learnt the hard way that you can't set a bandwidth limit when backing up to the kopia server, which can tank wifi performance on laptops during the backup.
I am thinking of switching to Restic as it seems like it is more suitable for people who want a tool at the complexity level of tar/git CLI with declarative config, rather than something imperatively configured.
> he Kopia server is convenient if you have a server at home and want to back up laptops to it, with them being in a shared deduplicated repository without having to trust the clients not to get pwned and wipe out the backups.
Neither can do it well. We just used S3 + versioning for that reason, worst case we would just rollback the bucket
[1]: https://github.com/rsnapshot/rsnapshot/blob/master/rsnapshot...
The best part: it is just files, not some proprietary archive. I have high confidence in this. I can restore with the most rudimentary tools.
This setup has worked for me well on linux, windows and now mac. I migrated to new computer&os by restoring the backup mostly.
For example:
Requirement:
"Just copy a file from folder A to folder B."
Minimal translation:
"Implement the capability to copy any file, in any format and of any size from SharePoint, located at a configurable path, with appropriate authentication and access control checks, then stream it in chunks to a different, configurable path inside an S3 bucket, also with appropriate authentication and access controls in place. Ensure that any disruption in either service which may occur while the file is streaming can be recovered from at the point it failed instead of having to restart from the beginning. Ensure that the retry mechanism is built-in and that the retry window is configurable; if the file cannot be copied within the specific time window, then an error should be sent via email to a configured email address. Ensure that the entire transfer is encrypted in transit... If the file happens to be a folder, then you must copy across all of its contents recursively up to a certain configured MAX_DEPTH to avoid DoS and ensuring that the system does not get caught in an infinite loop due to symlinks pointing to a parent folder... In this case, send an email to the configured address... Etc... Etc..."
And the thing is; if you tell AI "Just copy a file from folder A to folder B." - It will not meet your 'basic' needs because even if it does a great job at filling the gaps in your requirements, it will still take shortcuts. In order for an AI to avoid taking shortcuts, it would have to make you fill out a questionnaire and make you sign up for and configure services; it would not be a pleasant user experience. The user experience cannot be pleasant, because the AI cannot read your mind and it cannot know your intent.
"Just copy a file from folder A to folder B."
All that access control and authentication and symlink nonsense? That's a self-inflicted problem that exists only in enterprise, and shouldn't be assumed - much less created - for regular users.
Reality of software development is that the user would expect every single one of those things but they simply don't understand what they want up front.
There are three different aspects to consider in my experience; what the user says they want, what the user thinks they want and what the user actually wants.
The three can have substantial differences.
> Or you will complain if your colleague accesses your file without your permission.
This is a case of "out of scope", also security-first mindset that came from enterprise and is the inverse of how normal people expect computers to act. What if I actually don't mind or even want the colleague to access my files? I mean, that's half the reason anyone copies anything - to give it to someone else, or move it to somewhere where someone else accesses it.
What if the user actually didn't mean the files to be accessible to colleagues? Especially when the originals were, they just didn't know it? "Clueless user" problem is not something that should be solved at the level of filesystem operations. So either way, out of scope.
and a privesc also.
What I do is having a minimal systemd timer under root that only calls restic for backing up files (with some additional systemd seucurity restrictions). App-dependent backup logics, like dumping databases, are done by user/container-level cronjobs separately.
It would also be great if services did their own database dumps. Immich does this and it's such a relief to just be able to copy over its volume for backup purposes.
Unfortunately we don't live in an ideal world and so much of the software both doesn't dump it's own databases, and is incapable of running in a rootless container.
If you weigh the probability of a privesc from your backup with full access to the volumes, against the probability that the software you run has a CVE, which one is worse?
This is one of those things I consciously chose to ignore in my setup. Maybe it will come back to bite me in the future, sure. Or maybe I will learn a better approach that pushes my frontier of convenience-security forward.
Also why you sync through LAN only?
> tailscale or home routers network
I wrote about this here [1].
[1] https://staticnotes.org/posts/cloud-backups-with-restic/
We finally got a nice setup with CloudNativePG + Barman. This allows for point-in-time restores, but there were a lot of lessons to learn along the way.
- The various types of (database) backups (logical, binary, onsite, offsite, snapshots, write-ahead log...) in combination with the various types of data (database, files, cluster configuration...)
- In our earlier approaches, we tried to preserve the old database volume if it was not corrupt, and use that in our restore. This caused so many complications, because you are fighting the recommended approach. So now, when we need to restore, we always restore from backups and the 'live volume' is dropped.
- For a restore, we just spin up a completely new Kubernetes cluster, instead of trying to restore in-cluster. This is a lot easier.
- Many object stores allow for retention periods, which you can put to good use to prevent malicious or accidental removal of backups. HOWEVER, not all of them are really 'locked'. In some services, you can still override the lock with a forced delete; in others, you can still remove the project holding the storage buckets, which will delete the buckets, and so on... so test those things, instead of just blindly depending on a 'retention period' claim.
- We now automatically run a scheduled restore with verifications on a weekly basis. This requirement does shape your environment, so keep that in mind! There is also the question of how you can reliably and automatically verify that the restore restored the latest data (of a live prod environment). Various solutions exist here, but most are not very elegant!
Honestly, this is only worth it if you are already handling sufficient volume. If you are just starting out, then the easier approach is to just go with a hosted database, which will handle backups and point-in-time restores for you.
- do several backups (not using the same hardware brand or host if you do it online)
- check your backup (if something was wrong, you have another backup from the first rule)
- do not store your backups at the same place (if your place get flooded or burn, it will be pretty handy)
Using old standardized open source tools means it survives system updates without having to fix anything for years. I got my laptop stolen/lost 3 times over the past 15 years, and I have always been able to restore everything the next day on a new laptop, in the time it takes to transfer the files over the network.
I haven't given much thought, I'm sure there is a realistic scenario where this strategy would fails but I haven't found it yet.
What is missing is recovering accidentally deleted or corrupted data.
Yes but I don't use it precisely because that lets me recover files that I have accidentally deleted. I assume it's the intended default behavior of rsync.
The issue is companies love to fire QA people who should be the ones testing these.
Microsoft says we don’t need no SDETs, our products can ship full of issues.
What are the hostages, I mean customers going to do about it?
I have no sympathy for any major corp who suffers data loss. Now individuals who lost grandmas wedding photos, that sucks.
But my recovery strategy for that is simple. After scanning I emailed it out to one cousin who emailed them out to a few more. Plus I have a cloud backup ( I have no idea where the the original files are).
Data is meant to be shared after all
Most of other data i only have 2 backup, usually 1 at my local machine(code) and 1 online(github). Yeah it's not ideal, but another question to ask yourself, is it truly worth it. I trust the engineers at MS doing much better job at backing up their data than me.
I never though of daylight savings times causing such an issue. Learned something today.
However, looking at prices, I can see one of those backups just being dismantled to be used as more storage.
The sizes are all over the place because I only buy a new drive when I run out of storage and then just get the largest one within a reasonable cost. It has been cost prohibitive to mirror everything and current prices aren't encouraging. Neither is the rate at which HDDs get larger. I was kinda hoping for 100TB drives under $400 by 2025 back in 2014, but that hope has been shattered.
ZFS snapshots + restic backups to backblaze for my homeserver. My secret sauce is a healtchecks.io instance that blows up my phone if ZFS scrubs throw any errors, when local snapshots fail, or when restic checks or backups fail.
So who are we? A household with a localhost administrator?[0]
Then you are surely doesn't need encrypted, chunk-level corporate level bla-bla-bla.
You need a Syncthing copy to some other device not at home - and an additional backup procedure to maintain the history and protect against PEBKAC errors - which can run on your local Syncthing copy or/and the other one.
And no, nobody needs your 555GB of RAWs of sunsets/flowers/precious_family_moments you shot - not even you.[1]
[0] well considering the tone of TFA and most of the comments here
[1] that's Instagram/Google Photos/iCloud for nowadays, totally with "Remember this day N years ago?"
In my case, yes. I've gotten into the trap of building a homelab that over time became pretty important, more like a small prod environment. We take for granted the "stability" that cloud services provide (until the terms of service change and you are screwed over by either loss of privacy, loss of access, etc). Self-hosting these things yourself exposes you to this complexity.
Photos aren't that important, how about your passwords and documents and the whole setup around all this?
I also do a part of this just for the love of the game.
Then switched to Restic - so much better - highly recommend this.
don't be like me folks
If you accidentally delete something, how long does it take for the delete to propagate to all of your geographically separated datacenters? You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/versio...
uh no, in a company, as a sysadmin, you try your backup files at least every month
It’s simple
1) separate data from applications 2) use a real filesystem like zfs or btrfs 3) cronjob snapshot, replicate onsite, replicate offsite 4) monitor size of volumes and snapshots, delete them as your policy 5) periodic zfs and btrfs scrub
End of fucking story. No need for all that tricky shit y’all are doing.