Too Perfect a Mirror
jefferai.org
jefferai.org
Git repositories contain redundant information to perform consistency checks. If a bit flips in one of your repos, git fsck will immediately catch it. The KDE sysadmins thought these consistency checks were triggered when keeping a repository clone in sync and thus any FS level corruption would be caught at the first subsequent attempt to sync.
If you think "Gee, if I know this, surely they do?!" then perhaps the answer is: "They probably do and this issue is more subtle than you initially thought. Reread carefully before implying how dumb others were."
Yes, git --mirror should probably automatically invoke the moral equivalent of git fsck by default in order to catch internal repository inconsistency like the other git commands do, and the KDE team have been caught out by this. But they still don't have any protection against user error leading to loss of data with this setup as far as I can see, and that seems like a huge oversight.
The master repository is coded so that those types of actions are not permitted. Developers cannot force push or delete branches that aren't reintegrated into some other head, without assistance from the sysadmins.
After every routine push to a KDE git repository the new commit will be a descendant of the repo's original head.
With that in mind it's more understandable as to why they felt it was possible to take advantage of Git's design in the backup planning.
No "huge oversight" here, at least on this particular issue.
Only a backup is a backup. Backup, like security, is somewhat onion like.
However, regarding the backup question, an rsync backup would have been just as damaging to the anongit mirrors as git --mirror was. The whole point to git for KDE is that that "distributed" part of the VCS would help handle backups. And it has, that part has not really been in question.
The 'luck' came in where there happened to be an anongit mirror that was fully synced up so that we didn't have to crowdsource repo restoration, which saved a lot of time and anguish.
Had we at KDE ensured that the git repository being synced to an anongit mirror was fully consistent we wouldn't even be speaking about this: the git.kde.org repos would be shut down until the box could be restored and we'd have any of 5 easy backups to choose from to restore the repositories (the rest of the files on the box would be restored from the normal backups used).
I want to stress that this is the larger point here: It's possible in some ways to corrupt a git repository and have its subcommands not notice. You must use the provided git-fsck (directly or indirectly) before backing up a git repository, especially if you don't use git for the backup, or use git-clone --mirror.
The error wasn't that we weren't doing backups, the error was that we were making corrupted backups. tar | /dev/tape will do this to you just as badly if you get the right FS corruption.
COW snapshotting filesystems can help (if they have no bugs) but the KDE sysadmins were working under the errant assumption that git would make the integrity check in situations where that wasn't true, not that backups are simply not required.
Only a backup is a backup, until it's not a backup anymore.
1) Test your backups, to detect when your backups are no longer backups.
2) Make geographically diverse backups, so a single tidal wave can't wipe out your data. For bonus points, have enough geographically diverse backups that the world is probably ending if they're all being wiped out -- at which point you have bigger problems to take care of.
3) Make backups with a diverse set of mechanisms, so the failure or compromise of one (or N-1) can't fail and compromise all backup copies. Making backups on write-only media and hiding them means current failure or compromise can't fail and compromise previous backups, and may help back your data up against theft, landlords, angry neighbors, spurned girlfriends, or even the occasional corrupt government official.
Mirroring (be it software or RAID) is not a backup system: It is far too dumb, far too happy to overwrite your old good data with new bad data. You want a history, where old good data is not replaced.
Git is not a backup system: It is a version control system. While it may have some of the properties of a backup system as goals, that is not it's primary use case. As a result we see articles like this where we've seen how it can fail in achieving the goals of a backup system as a practical matter in this very article, even when intentionally attempting to use it as a poor man's backup system in the form of mirrors.
Such problems are not unique to git, of course. On a personal note, I've managed to wipe data with both git and perforce in moments of weakness. If you want to treat me kindly about it, you could say I used both to the point where the statistics were against me not shooting myself in the foot. And, fortunately so far, the use of proper, separate backup mechanisms have always allowed me to restore the majority of my data and left me relatively unscathed.
A backup clearly would have helped here.
This is the big thing I can't figure out what people are not understanding. git does consistency checking for you already, tar|rsync|etc. don't, so it makes sense to take advantage of that.
What we had was an instance of some of the underlying data becoming corrupt on the filesystem (with indications of that starting on Feb 22!). The big mistake was considering the source repositories as consistent and canonical at the remote anongit end, but the data would have been just as corrupt if we had scp'ed the repos from git.kde.org to the anongit mirrors around the world, since we would have bypassed git's internal checking in that way.
Is it safe to rsync a running mysql database at random times, or are you supposed to use mysql-provided tools to perform a backup?
Relying on consistency checks will not save you if someone
does the git equivalent of rm -rf on the master repository.
git checkout master
git reset --hard HEAD^
git push --force origin master
| New refs are pulled down to the downstream
| repository; all updates, including forced updates,
| are mirrored to the downstream repository.
| As a result, making a mirror clone essentially
| bypasses the safety checks in the repository
> If someone force pushes to a centrally shared
> repo, all hell breaks loose.
So that's an entirely different threat. On the one hand we have FS corruption, which is basically bound to happen and is likely to impact many repositories. On the other hand we have someone (maliciously) force pushing an update on a single repository that throws away all commits.
The first issue is one that needs a backup and restore procedure. The second is one that is at best an inconvenience and at worst a security problem.
Unless you want to include the the worst case scenario someone gaining access to the machine and force pushing such a destructive update on all repos. In that case restoring from developer copies may actually be safer anyway, because the machine and all its backups, be they mirrors or otherwise, may be considered compromised. That's a nightmare for which this discussion about backups is just quibbling.
Most service providers (who stay in business) have enough compliance features in place so that multiple authorized people have to be in collusion, a sufficiently senior executive must make the request, or there will be a "package" ready to be served to the client and relevant policing unit (police, FBI, whatever) so that charges can be brought against the malefactor efficiently.
While you may be limited in financial compensation by your contract with the service provider, it is absolutely in their best interest to avoid the situation with procedures (they do not want to be a party to a crime) and if those fail, to provide extensive records of the movement and disposition of those tapes.
It might sync broken objects, but it won't actually chose to destroy dangling objects without being asked to.
Human error is human error: sometimes people will do apparently egregiously stupid things if it's possible for them to do so, including things like deleting branches from a repository & then doing a git gc to "save space" shortly before realising that they weren't issuing commands to the shell they thought they were in. A backup strategy should be robust against dumb human mistakes as well as mechanical hardware errors, otherwise it's no backup strategy at all.
The same idea applies to the individual repos themselves. A git gc will delete objects unreferenced by the reflogs, eg deleted branches over a month old by default. They say that the total set of repos is about 25GB, so it would be reasonable to keep at least monthly backups for at a year or more.
The rest of that Wiki page is also pretty interesting for a general look at what devs can do on git.kde.org - server-side personal clones, personal repositories, a personal trashcan for repositories, etc. We don't have a pretty web interface for it (yet?), but it has some nice features.
What folks tend to consider the meat of ops work, often boils to a big ole boring checklist.
The problem is that you shouldn't just elect to skip a whole big section without some seriously good reasoning.
This isn't a slur on you or the KDE guys, hindsight is 20/20. I'm confident though that I'm not alone, that there are plenty of other Ops folks here who read the story and also felt the described setup violated a deep principle and just made feel ill at ease. These failure scenarios are not common, but the do happen often enough that we know to prepare for them.
As an example I'd point to how DBAs handle validation of replication - it's the same principle here.
Just for completeness, an example reason for not having proper restore procedures in place might be 'this is not the prime record copy of the data and it takes less than 24h to regenerate this data therefore this will be out of scope during restore tests'.
Who're clearly not under the impression they can't make mistakes considering TFA is a write-up of design flaws in a mirroring system :).
Look at it this way: If you think something obvious was overlooked, then it's good there's another report backing up your point. That's the value in everyone being open about their operations and experiences along the way - you only get better metrics for what works and what doesn't, in practice.
They didn't have that, or at least not recent enough backups.
Synching rather than snapshotting. It's a subtle difference (sync permits delete).
The system is designed such that the master repositories should never actually lose objects (even with force pushes and branch deletions, the admins make backup copies of the HEAD branch before letting those run so that the blobs remain in the repo).
As it turns out though there are repo tarballs generated periodically which would have served as a perfectly acceptable backup, and some other things the sysadmins could have done. The bigger shock for them was that git clone --mirror wouldn't actually run the git integrity checks (which they had mistakenly assumed).
But what you describe, is necessarily not a backup.
EDIT: redundant text removed
The thing that's missing is retention of old data, but I can tell you that is fraught with its own complications. A week-old repository tarball is almost worse than useless in the context of the git repository; we'd sooner restore that data by having a developer re-run "git push" than to lose a week's worth of development.
And that's assuming that a daily or weekly tarball isn't itself corrupt, which would have been the case here unless we ran git-fsck before making the copy (which is what was thought to have been getting run in some fashion in the first place).
I do fully agree that there needs to be more intelligence on the anongit side of the servers if they're to be used as viable backups instead of just sync destinations, but everyone keeps mentioning solutions to problems we don't have or null solutions to problems we actually have.
Despite what everyone seems to think we have multiple other backups of the source data (including tarball-style), but they're all crap in comparison to being able to recover from anongit.
The risk of using mirroring rather than versioned backups is that you lose all the data when a deletion is mirrored.
Redundant storage is not backup.
And even that is because of a deliberate decision on the sysadmins' parts based on a misunderstanding of how git clone --mirror responds to a corrupt repo, not some simple oversight. Which is to say, countermeasures will be put in place for that as well.
I do wish people understood why having relying only on even 2-week-old backups is unacceptable in the context of a large active FOSS project's source code repository, it's not like it's OK to simply start over again from KDE 4.9.4.
Yes, but i'm not convinced you see that this is EXACTLY what you are exposing yourself to.
What if next time it's a (nasty) bug in git? A push causes corruption perhaps?
Drop the idea of using git itself to host the backup strategy. Switch to plain old backups, if space (or performance - i'd wager the kde git repos must total a good few hundred GB, if not more, if there's artwork or other binaries in there too) is an issue there would be nothing wrong with incrementals for the */30 min backups.
Old-school engineering used to have the concept of a safety factor, where you used a material twice as strong as the one that your equations have shown to be strong enough to sustain the conditions you expect it to operate under. Richard Hamming also put this very nicely: "Would you fly a plane if you knew it depended on some function being Lebesgue-integrable?".
Was the scenario that something unwanted might get synced one day really that improbable or so unobvious nobody considered it? Hard to say in retrospect, but I would say no. Anyway, this might happen to every one of us, so I will use the time to reflect on the systems _I_ built and whatever assumptions _I_ could have done mistakingly.
The problem is that they were not stupid, they were too clever for their own good. A stupid sysadmin would follow the 3-2-1 backup rule and have on hand at least 3 backups of the data from the past 24 hours (or sooner, if backup strategy necessitated it). The KDE sysadmins and devs thought they were super clever and could avoid this rule from operations 101, and then found out the hard way that they could not.
This is really 2 separate discussions. One is the intricacies of git. The other is ignoring operations 101 by not making proper backups. Having over a decade in ops experience, I am more focused on the latter discussion.
As someone else here said, if they had followed a braindead checklist of how to do proper backups they would not have had this problem. Standard operations procedure will save you, even if you had made faulty assumptions about your architecture.
The main point is this: they broke an ops 101 rule. They had clever reasons and assumptions for doing so. These assumptions turned out to be wrong. Now people are arguing the point. It does not change the main point, that if they had followed the ops 101 backup rule, possible disaster would not have been knocking on the door.
You hit the nail on the head regarding overabundant cleverness. A lot of developers are too clever by half, too.
After I got to the "real world", I'm shocked that some extremely large companies (financial institutions that should know better) don't have anything like it in place, and have far worse and untested but expensive disaster recovery plans.
The number of times I've seen some sysadmin(s) base their entire organization on this faulty premise is absurd. Mostly it's because they have decided that RAID 1 or RAID 5 should be a decent "backup" strategy, but then there are those who believe mirroring systems is how to do backups.
They never, ever, take into consideration what happens when something corrupts/is deleted/is compromised. Without a way of going back in time (i.e. an actual backup) they are forever stuffed.
Sysadmins: MIRRORING IS NOT A BACKUP SOLUTION. STOP DOING THIS!!!
It's the --mirror option to much of git's plumbing, and it's not the same thing.
> Mostly it's because they have decided that RAID 1 or RAID 5 should be a decent "backup" strategy, but then there are those who believe mirroring systems is how to do backups.
I think he is implying that this is a case of the second. That is to say, I think he is saying that --mirror is not a backup strategy.
You're assuming it works like a traditional file system or block level mirror, but it doesn't. Corruption would in most cases have been caught. The weak (and accidental) link was relying on the server to give us a proper accounting of the current valid repositories.
We have established that he knows they used git --mirror, and I am pretty certain that you could not possibly know that he did not read the article.
You're better off asserting that your objects haven't changed (Which they weren't, and I agree that they should have been) and were valid in the first place (See above).
With snapshots, you'd invevitably want to dedupe them, which would be basically the same thing since it's append only, but with the dedupe infrastructure as another failure point.
The problem I think needs more attention here is ext4 silently corrupting data. ZFS has it exactly right with the built-in checksumming on write and read - it can't stop a disk going bad, but it can tell you exactly what's affected and _when_ - corruption would've been caught the moment the mirroring operation tried to read back bad data (and would've faulted the process, rather then happily return bad data).
Well, the KDE incident proves otherwise.
It would have been just as easy to push those corrupted repos to all of the backup tapes in the rotating snapshot set. A snapshotting filesystem could be a good backup (and seems to be what one of the sysadmins is pushing for).
But even more important is to fail fast and identify git repo corruption as soon as it can be detected so that further damage can be avoided.
The next two paragraphs identified two things that they weren't doing that they should have been. Otherwise they'd just have lots of snapshots of bad data.
Let me repeat what I said, italicizing the important parts:
Mostly it's because they have decided that RAID 1 or RAID 5 should be a decent "backup" strategy,* but then there are those who believe mirroring systems is how to do backups.
It seems that an even bigger design flaw is that they (still) aren't doing regular backups. The mirroring of course provides some redundancy, similar to what raid does, but as they say: "raid is not a backup solution".
Plus unchecked backups of corrupted data aren't worth a lot, and corruption-proof mirroring acts as further (and timely) backup.
I read the article same as the OP, it doesn't mention any backup system, only the mirroring. In fact, they did restore from a replica that was out of date in the end (from projects.kde.org). Had they had actual backups, the article would surely mention they only used projects.kde.org because it was somewhat more recent than their last backup?
The story about planning to do regular ZFS snapshots hints to the same, if they had a backup system they wouldn't need that.
edit: sorry, is that your post/do you have more insight? In that case I'm sure you know better than me speculating on what the author meant ;-)
Nah, I'm not Jeff. I have some general insight because I was involved with setting up our git infrastructure in its early days, but I haven't worked on the mirroring code, and I've been out of the loop on day-to-day admin operations for a while, so I can't comment on the backup schemes that may or may not be in action on the servers right now.
But loosing one weeks worth of work is much better than loosing 10 years worth of work!
However, while we planned for the case of the server losing a disk or entirely biting the dust, or the total loss of the VM’s filesystem...
Any backup strategy with only one backup (instead of preserving historical snapshots) is vulnerable this way.
Even if all the official repos where destroyed, all they'd have to do is ask the last person who'd pulled to give them a copy of their clone.
No doubt it would be a pain to do, but no data should have been lost.
As Linus said: "Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it"
Linus Torvalds once coined an adage that "real men don't make backups. They upload it via ftp and let the world mirror it." Well, the FTP bit isn't true anymore, but otherwise DVCSs have enabled this for mere mortals.
Between them and the large number of developers who would have copies of the repo, reaching consensus on what the "true" repo was - while not easy - could be done in a secure fashion due to the hashes. You wouldn't have people declaring "no it's totally it" and not being able to verify.
Also I would be interested to know why KDE don't use something like GitHub or BitBucket? It would be cheaper for the organisation, and they could still setup web hooks to get notifications of commits.
That said, we did originally try to work with Gitorious, but that didn't come to pass for a variety of reasons. Here's a small writeup of our journey back then, some of the parts discussed there have since been swapped out or fleshed out further: https://news.ycombinator.com/item?id=2972107
Because no, the data isn't all in git. For example, which set of credentials was used to write which piece of data to a repo isn't stored in git, it's logged by the authentication system in front of git.
This is in fact the main selling point of DSCMs - everything's a repo, repositories can (provided access) exchange data bidirectionally, and data can take all sorts of interesting routes from A to B involving as many repositories inbetween as you'd like.
Negotiating the actual write access to a repo happens entirely outside of git. That's really a good chunk of what software like Gitorious, GitHub or gitolite do, and then on top of that you get into the collaboration features and stuff.
It may not be kosher, and I don't think the porcelain supports it, but I think you could write a remote that only accepts commits if they are all signed by someone on a trusted list. This would probably mess up a lot of workflows though, since if person A writes some commits, person B rewrites them to sign them and then pushes them to the repo, then when person A pulls the commits "they" created they would actually be different commits.
If we were willing to take that hit though, I think it should also be possible to modify git to permit multiple signatures on a single commit. Users could take a signed commit, append their signature too it (well, technically not append. Signatures are in the middle it seems) and then push it or pass it on to others to sign. This would only allow a single commit to go through the system at once (since commits after the first would need to be rewritten, invalidating previous signatures) but on the other hand it would let you create an integration server that only accepts commits that have been signed by some number of trusted code-reviewers, while still keeping all of that information in git. So long as trusted code-reviewers all signed off on it, who actually sent it to the integration server probably would not be particularly important. Such a modification to git would probably be extremely un-kosher of course...
As for Hetzner: The git.kde.org master server isn't at Hetzner, nor are a bunch of the mirrors. Our infra is pretty distributed and eclectic as far as hosting locations go, partly because a lot of the resources are donated from all over. We don't "support" any hoster in particular.
A SYNCHED COPY IS NOT A BACKUP.
This includes:
- git
- svn
- Dropbox
- RAID (of any kind)
If you don't believe this, please reconcile everything you've learned about backups. More specifically, if you treat any VCS or dropbox as your only backup system, STOP RIGHT NOW and at least get something that's intended to be a backup, such as SpiderOak in backup mode.
The mirroring system which git provides?
They used the safe version before, because they were running into problems with the integrity checks, i.e. ref deletions and non-fast-forwards.
What they should have done was to write a hook or a script that did those non-safe updates manually (maybe only for some repositories, and some refs, don't want to rewind e.g. master).
But instead they completely bypassed the safety mechanisms and got screwed by corruption.
> our mirroring system had bugs
> Originally, mirrored clones were in fact not used, but non-mirrored clones on the anongits come with their own set of issues, and are more prone to getting stopped up by legitimate, authenticated force pushes, ref deletions, and so on – and if we set the refspec such that those are allowed through silently, we don’t gain much. A hybrid approach of a non-mirror initial clone followed by a shift to mirror mode could force the server to validate the entire repository as it packs it, so that is something worth investigating.
Going back through years of backups to find a non-corrupt copy can take a lot of time during which your service is down. Not a perfect solution by a long shot. Discovering which files have been updated and which are corrupt is also non-trivial.
The reason I thought of DVDs is that they're not sensitive to electromagnetic fields as disks and tapes. (You never know: http://www.telegraph.co.uk/science/space/9097587/Solar-flare... )
Not sure why it's not used "more". The tool is pretty straightforward.
My backup strategy is as follows: my most important files are in my Dropbox folder. So they're both on my computer and on Dropbox' servers.
But what if my drive goes bad and pushes corrupted files to Dropbox?
That's why I have another client with Dropbox that I only turn on every week or so. I hope that if something goes wrong (including Dropbox itself wiping all my files both remotely and locally), I can still get the older versions. That, and time machine backups (they include Dropbox folder).
Git mirroring is great, and periodic checks for consistency would help, but snapshots taken and stored (offline) for reasonable periods of time are the only reasonable backup model. There are corruption issues, availability issues, etc. where offline backups are far more reasonable. Ideally you would separately cryptographically sign your backups (which is easier than just keeping track of hashes), too.
(and obviously a backup system is meaningless if you don't also check for restores periodically, and monitor the success of the whole process)
"I’d love to see this in use, but, after having had excellent experiences with it on Linux for a couple of years, I’m a ZFS fanboy at this point; and, I don’t know how well it’s supported on SUSE, which is the distribution git.kde.org is running (although I’ve run it on Gentoo, Debian, and Ubuntu without any problems)."
For admins there is another interesting one: they artificially generate an extremely vulnerable spof - and after this desaster hey are still doing it.
Can you see the mistake?
Hint: if you have many things, why do you make it one thing?
Of course I am willing to help - the problem is desribed clearly in the first point of the author: they are generating one "projectfile" - whatever this looks like, it is a reduction of many to one. The distribution of 1500 git repos with n-thousands of files is relying on one single file - there is no technical need for that, in fact it eliminates the power of distributed repos by reducing reliance into the presence and integrity of one single text file.
The author writes about the process of this file beeing corrupted and triggering a random repo killing process - the incident is a good bad example for what can happen if you anticipate the antipattern of making one out of many.
Building redundant systems you always try to achieve the opposite - make many out of one, to eliminate the spof. You can not scale infinitely with this, because in the end we are living on just one planet.
However, unneccessarily making one out of many is the wrongest thing you could do building a backup or code distribution system. This antipattern still exists in many places and should be eliminated.
This is not about filesystem corruption etc. - the reason for the destruction was one single project file. Do not do this. It is not critical for a backup system if it takes long time to scan a filesystem for existing folders over and over again. A backup system is not a web app, where it might be a good thing to do one out of many (aka as caching in this case), but a backup system does not need this reduction.