The Classic Unix Horror Story

I did something similar once, but realized pretty quickly I had been in the wrong directory when I hit return after "rm -rf *" - I cannot remember why I realized so quickly, it's been over 20 years, but it seems to me I had multiple terminal sessions going (probably via screen) and freaky stuff was happening in one.

If I remember correctly, I was in /dev when I meant to be in a sub-directory I'd created for testing purposes. Some of the freaky stuff included losing command echo, e.g., since my TTY no longer existed.

Fortunately, since my work involved backups, modems, and a few other things, I had a pretty good grasp of major and minor numbers and was able to recreate my TTY using mknod. After that, panic dropped considerably and I could at least poke around and figure out exactly how borked the system was.

Another time, a colleague and I spent hours trying to restore from tape to a filesystem we were pretty sure was corrupted. We tried every trick we knew - and learned a few more along the way - but eventually, at least in part because we were stressed out and sleep deprived, we had something along the lines of the following conversation:

"The FS is borked, right?"

"Right."

"We're confident the backups are good, right?"

"Right." (We'd built the backup system, it was in use on hundreds of servers across the company, we'd never had a failed recovery....)

"So let's reboot into single user, newfs the disk, restore from tape, and boot back into multiuser."

"Uh..."

"I mean, we cannot possibly make things worse, right? This machine is thrashing."

"Yeah...."

"So?"

"OK."

It worked. When we returned to our senses, we realized we were lucky it had, realized we were deluding ourselves (we could have made things worse), and realized that if it hadn't of worked, we would have been fired in all likelihood: We'd made a decision way above our pay grade, as it were, with no management knowledge at all.

The next day, as far as they were concerned, we'd spent hours slaving after midnight to save a build machine and we kept the design group working. We just smiled, awkwardly.

Oi.

Nowadays you could sync and then force a shutdown (hard), then mount your drive on another machine. A hard thing to do in the days of vaxes.