At one place I remember that we had the usual scheduled backups for our MySQL databases. We also wanted a quicker way to bring systems back with less data loss, so we also had a lagged replica that purposely applied changes delayed by I think it was about 4 hours. The other handy thing was that swapping a replica for the write leader was a common operation that we did whenever we upgraded hardware or software versions so it was a well exercised procedure.
Lagged replication is truly a great feature. Nowadays, on most systems we run one instant replica and one with 24 hours delay.
W. Curtis Preston, author of the O'Reilly book "Backup and Recovery" has said:
> Backups are useless. Restores are priceless.
I also like:
> The state of a backup is unknown and unknowable, until such time as it has been tested with a restore.
Sounds legit. This goes double on the cloud. In my early days of using AWS, it was so easy to create and destroy EC2s, I created some EC2s for testing - then terminated them... then realized I terminated the wrong ones.
Whoops
human error is here. creating a backup and forgetting the password for example or mistyping it
so true, had a zfs nas in 2009 with raidz, snaphots, everything, yet I did delete personnal stuff by mistake.