At the time, the main thing hosted on that machine was my WoW guild's website, which I had been working on for close to a year, and was beginning work on converting the site over to a general purpose guild hosting site.
I was doing some work for a client, setting up a mirror of sorts for some kind of yearbook thing I had built for them. For that, I made a script that would mirror the online yearbook with wget, zip up the whole directory, then clear out the mirrored pages (all I cared to store and serve was the zip file).
All of my websites were stored in /www on the server, and the raw yearbook was located at /www/clientname/www. Inside the clientname directory, I had the mirror script which was something like this:
wget -whatever_options http://whatever.address www
zip yearbook.zip www
rm -fr /www
My heart sank when I tried loading my guild page a few minutes later (just to see what was going on on the forum, if anything), and it served up a bunch of 404s.
I went to /www/guildsite and saw it completely empty, and almost immediately figured out what had happened. At that point, I had to get my composure and figure out what I was going to do (I did not have backups or source control). I unmounted the directory, and went off to lunch with a friend, shaking with anxiety.
Upon return, I started writing out a perl script to scour the device byte for byte looking for PHP scripts, looking for instances of <? and then capturing the following 2000 lines or so, assuming that would be a sufficient buffer. When the script was done, I set it to run on the partition, and 45 minutes later I had a few hundred files to work with.
I had to manually go through every file (which were named numerically in the order they were found in the filesystem) and determine if it was the most recent (or recent enough) copy of the code, clear off any trailing bytes, and rename it to the filename it used to have. Luckily I could remember the name of almost every file in the system. It took about 8 hours to go through the few hundred files and recover them.
Needless to say, I learned my lesson after that, but the feeling of victory I got from recovering those files from the device was epic.
6 years later, I realize that that's a rather trivial thing to do, but at the time, I didn't know what I was going to do, and remembering that the file system doesn't clear all the bytes of the a file just it's reference gave me tons of hope.