Another true story: A while ago I was DBA on a very important database (as in, the company would likely go out of business if it was lost). The way it was backed up was to quiesce the standby, take a snapshot of its filesystems on the SAN, then send those to tape while the standby was reenabled. Sounds good. But what had happened was that the sysadmin, to speed it up, was running several parallel backups, each one of which had a hard-coded list of files.

So as soon as one new datafile was added to the database (I estimate, a week after that regime was put in place) all subsequent backups were invalid. We were in that situation for over a year and everyone believed that the databases was being properly backed up. We would even occasionally restore a file (picked from the backupset, hmm) and block-verify it. I only discovered it when I was upgrading the system and I fixed it real quick (dynamically generating n equally-sized lists of files is trivial!). Hard coding the list of files was stupid, sure, and betrayed a fundamental lack of understanding about how the system worked - but it was also stupid that no-one ever checked, and that betrayed a lack of understanding of how organizations work.

The lesson I took away from that was, you have to have people who have visibility of things end-to-end.