Using a bash script I wrote and running as root calling rsync to backup to /mnt/backup/blah which was supposed to be an nfs mount.

At some point [turned out to be 5 months prior to attempting the restore] the nfs became un-mounted and I was backing up to a dir in the same drive in /mnt

Yes the hard drive had died.