Unix recovery legend | Hacker News Reader

Unix recovery legend

ee.ryerson.ca

220 pointsaraneae83 comments

I got into this situation once. I got pretty far into my recovery before being stymied by the discovery that tar is only distributed via tar archives: http://www.gnu.org/software/tar/tar.html

Is a "shell archive" also a tar-archive?

Nope, nor is a cpio archive, both of which are available for download at that site.

I have experienced this very thing with a SunOS box at the remote end of a satellite link. It was in the US and we were in NZ. Let's just say that you can do a lot with bash and a single root shell.

I would have hit Control-D just to get the free trip to New Zealand :)

You might earn a trip to the unemployment office instead.

Wasn't he already in NZ?

I had a similar thing happen (having to recover from a very very stupid mistake), I hadn't deleted everything, but I had moved the entire tree into /var/www (oops)

This is an profanity-laden blog I wrote immediately after I figured out how to fix it and got everything back up and running:

http://newslily.com/blogs/94

(I'll be the first to admit that I sound like an obnoxious teenager in this blog, and that some of the things that I tried to fix it were very naive...it was a few years ago that it happened and as stupid as it is to admit, trying to get everything back up and running made my adrenaline pump pretty hard, so...I apologize in advance...)

To be fair: you had this happen on a piece of consumer hardware which can boot off of any of several dozen live CDs or USB drives, available off the internet at megabit speeds using one of the (no doubt) several other pieces of consumer hardware you have in arms reach.

The VAX 11/780 had no portable boot device, and BSD unix had no "installation media". You bootstrapped to a system monitor, loaded a second stage off of tape header into memory manually, and hand-build your root filesystem. And that's assuming you still have your original tapes from Berkeley in storage somewhere.

I actually tried doing this on the simh simulator a while back, out of sheer curiosity. I gave up before managing to make it work.

I wish! This was done on a VPS located god knows where and run by a company that was owned by people who didn't seem to speak any English. I remember emailing them and asking them if they could just move my dirs back using a chroot, but they had no clue what I was talking about. If I had allowed my SSH session to time out, I would have most-certainly lost the server and everything on it.

I have since switched to a combination of slicehost and linode.

... I would have most-certainly lost the server and everything on it. I have since switched to a combination of slicehost and linode.

Switching to a different VPS provider is not the right way to prevent data loss. Doing backups is the right way to prevent data loss. :-)

This is true, but he gets props in my book for being smart enough to switch away. It seems surprisingly common that people don't do this.

Absolutely, and I didn't say that switching VPS providers was a replacement for backups (the linode and slicehost boxes now trade rsyncs every night which include mysqldumps). My point was that I think if a similar thing happened now, that the fine folks at either provider would be more able to help me out.

Even still, I would much rather be able to just mv the dirs back to their proper location from a chroot than having to do a restore from scratch over the network.

It's one thing to work under root sometimes, and it's another thing entirely to work in the root directory. I'm still trying to understand how the file upload sent a bunch of images into the root directory. I would undoubtedly have scp'd or ftp'd the files to either my home directory or the standard ftp directory with user permissions, and then I would have moved the images to where they belong (as root if necessary).

Still, you had some good points. The information on explicitly indicating the library directory is good. Most important of all is the lesson to double check any mv, rm, and splat-completing command that is run as root (or under sudo, for you sudo junkies out there). Personally when I'm using a * in a root command, I first add "echo " to the beginning of the line so that I am sure of what is going to happen before I execute it.

I was being incredibly lazy and did the following:

    pscp *.jpg root@gibsonandlily.com:/

This was monumentally stupid, I know. Looking on the bright side of things, however, I learned my lesson :).

You know that if you just left the slash off, the files go into root's home directory (typically /root/)?

On a separate note, I tend to disable root SSH access on my servers. I SSH as a user and then when necessary, I'll su to root. Not only does that kill the temptation to use root as the default login, it also makes it doubly-hard to get to root as far as password-stealing goes.

Single most important habit I've developed as a SysAdmin: measure twice, cut once.

I never just

   sudo rm *

First, I do

   echo sudo rm *

inspect the output, and then use CLI editing to recall the command, and delete the "echo"

I do this for damn near everything that involves a wildcard. It's a lifesaver.

The rm still makes me nervous. I always just do "ls *", then replace the ls with rm when I'm convinced those are the files I want to delete.

I have added a like

alias rm='rm -i'

to my .bashrc. Sure its a pain when i have to delete whole directories but saves my ass.

this is default profile on almost any mainstream linux distro these days anyway

Not on Arch

with tcsh, "set rmstar" will give you a confirmation prompt when doing something like "rm (star)".

and while i'm at it, "set noclobber" has saved me a few times from doing something like "grep file > someotherfile" when i meant to do "grep file >> someotherfile".

I always add the absolute path to the star, so I never accidentally issue rm * down the line when scrolling through the command history.

More fun: libc is hosed on a remote box and you can't find any working statically linked binaries with network support, but still have a few live shells accessible.

I was starting to write a UUDECODE function in pure bash (I'd saved my ass before by pasting uuencoded binaries into terminal emulators) — but then I remembered that bash has built-in tcp support!

At least when called from the exec builtin to spawn a file descriptor, you can open /dev/tcp/$host/$port psuedo-devices. It's existence would have saved me countless more times, if the spoilsports at Debian/Ubuntu hadn't disabled it at compile-time.

> but then I remembered that bash has built-in tcp support!

You have to be careful making that assumption anymore, though; the version of bash that's included with Debian (and downstream variants, e.g. Ubuntu) has /dev/tcp and /dev/udp support disabled.

So it's not something you can count on having available in any random environment you log into.

I disagree with Debian's decision to disable it, but they seem pretty firmly set on it -- people have been arguing with them to reverse it for the better part of a decade, to no avail, so I doubt it's going to change. It's a declining minority of users who probably care. But you could do some neat 'party tricks' with it.

zsh's built in zftp support, which can be loaded and run without a working ld.so, has also saved me at least once.

This should only happen to you once in a lifetime.

and this is why my "rm" is replaced by this script... just moving stuff to another directory (trash) and I clean my trash manually (with at least 2 confirmations)...

With all these precautions, if I delete something important, I don't deserve admin rights :-)

randstring=`date "+%Y-%m-%d_%H-%M-%S"_bak`

while [ ! -z "$1" ]; do mv -b -S "$randstring" "$1" ~/.local/share/Trash/files shift done

-hbt

I wish there were an idiot mode for unix that would tell you exactly what the command you just entered is going to do. I'd be turning it on all the time when using commands I rarely use.

It would be nice to know I'm about to do something that I don't intend. As long as the system doesn't ask me "are you sure you're sure you're sure?"

Redhat's default bashrc aliases rm to rm -i. The i stands for idiot. (But of course, -f overrides -i, so maybe it's actually a wrapper. Who knows.)

What we really need are transactions or a versioned filesystem. rm -rf *? rollback!

Yes, undo rather than confirmation is not only a good idea for GUIs, but also for CLUEs [1].

[1] Command Line User Environment

Command line mysql client has a i-am-a-dummy mode where you cannot run a query like "DELETE FROM table". You could of course run "DELETE FROM table WHERE 1=1" but at least it saves you sometimes.

This is awesome, I never knew about this; I usually type "delete from where" and then fill out the query, just to avoid accidentally hitting enter after typing "delete from users".

For me, this is sort of like standing at the edge of a big cliff; you just have this weird feeling that you might accidentally, compulsively jump.

Man, I thought I was the only one who felt like this (the cliff thing). I told a friend once and he became worried that I was suicidal. (I'm not.)

I get that too :)

WHERE 1=1 won't actually work when i-am-a-dummy mode is enabled. You must reference at least one KEY column in the where clause.

Well, my point is that there are ways around it. Like WHERE id IS NOT NULL, or WHERE id=1 OR 1=1. It's not full-proof, it just protects you from writing out a query that is not complete.

It sort of sucks that in SQL you first say what action you want to perform and then say what you want to perform it on. I end up writing the WHERE part of every query first anyways.

In C# query syntax, you write in this order:

  from where select

I like it that way; it happens to already match the way my mind plans a relational query.

Also useful:

BEGIN;^M<potentially dangerous query>;^M<look at results>[COMMIT|ROLLBACK];^M

Zsh does that for you:

    $ rm *
    zsh: sure you want to delete all the files in /home/viraptor/tmp/zsh [yn]?

Nice: also prompts when using the -rf options.

If any of you ever have the same problem, here's the hex for future reference:

Bookmarked! :)

-1? humor is allowed on HN you know :)

You work on a lot of VAXs? That's VAX assembler.

You never know!

(we actually have a huge pile of old old machines in a store room at the office here - Im not sure if there is a VAX in there or not.. there could well be)

I wasn't being too serious though :)

My favorite recovery (though it was not fun at the time) involved a headless machine that wasn't booting, and all I had available that would boot on the machine was a floppy disk that ran a rescue shell. I modified the floppy to start linux with console=/dev/fd0, and so was able to type blind and then remove the floppy to read the output. After a few painful cycles of this, I noticed I could often guess at what was happening by listening to the write patterns of the floppy drive, and completed the rest of the recovery using essentially morse code.

I stopped $ rm -rf ./things -and instead do- $ mv ./things /tmp and just forget about it. Yeah yeah, it won't work on a server, but for a home system that reboots occasionally, it should work fine.

http://code.google.com/p/trash-cli/

I've learned that nobody and nothing should use /tmp, on account of it being 777. The idea is that Mallory can set up a symlink from an anticipated filename ("things" in your example) to an important file or dir. So writing to /tmp/things destroys the symlinked file/dir.

Of course if they had been running the proper operating system with a proper filesystem on the Vax then this wouldn't have happened.

You unix kiddies finally got a versioning filesystem now I hear ?

Reminds me of my first job straight out of University (1988) - there was a client who had a sysadmin who had a bad habit of logging in as root and running rm *. This didn't cause any immediate problems - but all hell broke loose when tried to reboot their system.

They had about 25 users on a 386-25 - quite a decent system in its day.

Does someone know what's this use of grep as cat he alludes at ?

You can use

  grep '.*' > /etc/password

instead of

  cat > /etc/password

since echo still worked, why would you need grep for that one?

echo, as he says, was built into the korn shell, whereas grep was a binary he deleted from the filesystem.

Because it does something entirely different. "echo " is the same as "ls", while "grep '.'" is the same as "cat".

Match any non-empty line.

   /usr/bin/grep ".*" /path/of/file

Sod any 'non-empty line'

  /usr/bin/grep '^' > output

Or even better, grep can be an editor with line filtering - if you make an mistake, have it throw away the line:

  /usr/bin/grep -v '%' > output

(Any line with a %, throw away, keep the rest - good for multi-line files)

That command actually matches any line. You seem to have confused it with egrep ".+".

Perhaps using grep to match anything and thus output the whole file?

    grep '.*' file

What if you were to symlink your entire tree to say /z - as far as I'm aware, this would not use much more disk space, and as it would be the last to go, could you not call /z/bin/whatever ?

I may be way wrong here, but I thought the symlink would still work with the original being deleted, because the file system won't actually mark the file as gone until all links are gone.

Would (or does) this work? Haven't got access to a unix box right now, so can't try myself, but thought it might be a naive failsafe (at least until you actually wait for rm -rf * to finish...)

Symlinks won't save you but you could use hardlinks to make two pointers to the same file. You would have to do it for each file — was it cp -l which would do that.

On Linux, I've used "cp -al" to make a forest of hard links, consuming little space.

That's called a link (a hard link). It's just another reference to the inode on disk, indistinguishable from the original. All regular files are hard links, most just have a link count of 1.

Symlinks are symbolic links (soft links), potentially-relative references to fungible user-facing filesystem paths, that work across filesystems and respect the permissions of the original. They're a special kind of file.

It would work with hard links, but you'd have to make and maintain a whole separate real directory structure with an individual link to every file. People used to do this to efficiently maintain branches of the Linux kernel in the days before bitkeeper, depending on their editor to replace files by rename+copy instead of rewriting in-place.

Hard links to directories were removed in an early unix for being hostile to reality and fsck, but were added again in Mac OS 10.5's HFS+ implementation to implement Time Machine's storage.

  # cd 
  # rm -rf *

I'm wondering why they can delete files under /bin and /dev?

Because they are root and those are just files.

Perhaps they need to alias rm to 'rm -I' (-I : prompt once before removal)? I remember one of the MandrakeLinux versions had that enabled by default many years ago. It saved me a couple of times when I was still a Unix noobie.

"Because they are root and those are just files."

Just because you are root doesn't mean you should be able to do whatever you like to the filesystem at any time. Most modern unix variants (including linux) support file attributes such as immutability, see the man page for chattr(1) if you want more. In BSD land there is additionally the concept of Secure Levels which limit the ability of root to change file attributes under different circumstances; generally you have to reboot into single user mode to overwrite files or directories marked as immutable or delete ones marked as append_only.

According to the FreeBSD chflags(1) manpage:

  The chflags command first appeared in 4.4BSD.

Which puts it about seven years after this story.

"4.4BSD" isn't the same as FreeBSD 4.4, but actually version 4.4 of the original BSD which was released in June 1994.

OK, eight years, then. The timeline I found gave 1993 for 4.4BSD.

AFAIK a single cd command will set current working directory to /root, and rm -rf will only remove everything under /root. Or does cd command had different behavior in Unix at that time, comparing to shells today?

That sort of thing is why /root is now root's home directory.

The homedir of root was probably / on that system.

Did that version of BSD have secure levels? Updating systems where the system binaries are marked immutable can sometimes be a pain; but the comfort level it provides is remarkable.

Awesome story. Thanks for the repost. I've never read this one.

I appreciate the filesystem snapshots that ZFS give me.

Fairly certain that zpool and friend live in /bin or /sbin, so you'd still be pretty hosed if you lost all your system binaries.

While true, you can boot from a removable media and restore the snapshot fairly quickly. The original article author seems to have had a problem where he wasn't sure what exactly did the system/boot backup contain.

The choice of what snapshot to mount is part of the boot process, as chosen by grub. Nexenta's apt-clone creates a new snapshot for every system upgrade and adds a new boot entry for rolling back in case it doesn't work.

if you could recover "cat" and if there was /proc.. oh well, lots of "if there were".

Perhaps just use shells with more builtin commands? Something like busybox. So if you ever do rm -rf * and don't close your shell you still have all the basic utilities available.

Busybox is a binary that inspects argv[0] to determine how to behave. This is very different from shell builtins that do not launch a new executable, and will thus function correctly after the executable is remove from the file system. As far as I know, you cannot get busybox to perform it's other functions as "builtins". But if you have a functional debugger, it should be possible, though computing offsets is likely to be hard without another machine (chances are that the busybox binary has been stripped).

While busybox is running, if /proc is still there, you could probably use its file descriptor to get it back.

while a newbe i asked on irc what would happen if i move /usr in my sgi indy to a new scsi drive i just installed. sice no one replied i opened a terminal and `sudo mv /usr /newusr; sudo ln -sf /newusr /usr` or something dumb like that.

While i waited i opened netscape and went to read something. Then while it was coping irc was flooded with bad omens and curses about what would happen if i did that on a live system.

The only thing i noticed was that the icon state (the desktop shortcuts changed picture if there was any process open for their target files) would some time blink. But that was it.

Not a thing happened. Even the bookmarks i saved on the netscape bookmark file opened before the move ended up on the right place.

Reply on news.ycombinator.com