When a node went down there was no hope of ever coming back up unless you shut the other nodes down as well. While this was going down of course none of our ingress data was being inserted so it built a queue. When we turned things back on the queue would overload vertica again and we had to repeat the whole thing.
Fortunately for us we only stored analytic type data on vertica where customers usually were only interested in the last few hours anyways. So we ended up deleting all historical data and just reprocessing it over months while occasionally prioritizing customers that complained.
Gray•Duffy, LLP Settles Two Massive International Data Loss Claims Arising From Computer Server Failures
Arc Touch, Inc. vs. Atlassian PTY, Ltd
http://m.grayduffylaw.com/?url=https%3A%2F%2Fgrayduffylaw.co...
https://www.theregister.com/2012/05/09/atlassian_cloud_stora...
In my limited experience these diffs can be missing information. I recently had to reconstruct an issue description using these email diffs after two people where editing the description at the same time and it was not 100% accurate, several lines were missing. Going to the 'history' tab on the issue I was able to get the missing lines however, if all you have are emails though you might be out of luck.
3-2-1-0 Applies to all data, at all time, in all places
3 copies
2 different formats (ex. HDD, cloud)
1 offsite
0 lost data? :D
That's not what "format" means, it's more like DB-Dump and DB-VM-Dump, or pure Files and VM-Dump, or something like restic-repo and rsync(pure files).
The idea is to not have the backups stored on the same hardware or even same type of hardware. Same hardware is obvious but same type of hardware is listed because if a manufacturing defect or a known vulnerability is present it would make all of your backups at risk. So you want to have backups stored on 2 desperate types of storage media. HDD and Tape, or Cloud etc...
As far as running veeam on esxi, you would need to elaborate more on that
I had some backups that where not backwards compatible from versions that worked with ESX but NOT on ESXi. There's your "backwards compatible".
>As far as running veeam on esxi, you would need to elaborate more on that
Yeah you know exactly what i mean, no need to elaborate on that.
wow, ESX, I have not seen anyone talk about that for a long time, you must really hold a grudge...
I never used Veeam with ESX so I can not comment on that.
>Yeah you know exactly what i mean, no need to elaborate on that.
No I really do not, you do not run Veeam on either ESX or ESXI, veeam connect to vmware with API calls, so .....
I made efforts to buy HDDs from different sellers even, to avoid sequential failures from singular bad batches. That's something else I'd want to add to a "3-2-1", with regards to HDD as a form of backup or storage media.
Technically alot of Backup Planning has moved to 3-2-1-1-0
3 Copies
2 Different Media
1 - Offsite
1 - Offline / Immutable
0 - Errors from Verification Tests
My 3-2-1 comes from a personal non-professional standpoint, thus not having the extra 1-0. However I have been considering immutable offline backups, using burned DVDs or Blu-Ray discs. That's another project for another time though, for now I'm trusting paid cloud providers.
As for verification tests, hashsums are a simple solution in my opinion, but I've moved to ZFS and BTRFS to avoid having blips.
They're almost certainly rebuilding something from scratch.
If it were an AWS system limitation, almost all of those can be lifted if you ask nicely and are a big account.