Unix Admin Horror Story Summary (1992)

I was in a department running on a mix of Wyse green-screen terminals and, later, X terminals, when we got a budget upgrade that would roll out actual individual PCs -- 486s running SCO Open Desktop -- to everyone. (This was not cheap, it cost about £4000 for the hardware per seat, although the software was free because, er, this was back in the day when SCO was a respectable UNIX development house rather than a serial litigation zombie, and we were SCO's techpubs department).

Anyway, the editors, who were techpubs management (and thereby stronger on the management than the tech side of things), got their workstations before anyone else. And one of them thought, "ooh goodie, my very own UNIX system!" And proceeded to do "sudo chown -R me:me /" (substitute their username and group for "me") all over the root filesystem.

It's amazing what breaks when every shared library suddenly belongs to a random user, isn't it?

...

Except that what he did delete, it turned out, was the actual organizational structure of the Active Directory tree, including ALL user accounts. (It's hard to explain without visuals aids, the UI gave no indication it would delete the actual AD objects, not just the (non-existent) GPOs.)

Before long, people started calling to let us know they could no longer log into their computers or the terminal server. sigh It was a fairly stressful morning.

We really tried, for about 45 minutes, to resurrect the Active Directory tree, but it was no good (this was Windows Server 2008, so no AD Recycle Bin), so we had to restore the server from backup. I have since learnt that there is backup software that allows you to restore, say, your AD tree, or maybe even just a part of it, instead of the whole machine. Well, the backup software we had at the time suuuuucked, so not only did we have to restore the entire server, but we had to literally sit all day and watch the progress bar move at glacial speed.

In the end, we had the server up and running again, and fortunately both the company's CEO and most employees actually welcomed the opportunity to finally, FINALLY clean up their desks, something every single one of them had been delaying for a long time. And by the time we were done, I was just so exhausted I wasn't even mad at the newbie anymore.

At least we learnt from that mistake, though. Got ourselves a second domain controller, and a much better backup solution. In retrospect, I think it was probably a good thing - our boss took it with good humor, no data was lost, our backup system worked, but we also saw how badly it sucked, and the incident gave us some leverage to get the funding for said upgrades. Also, everyone had a clean desk, and since it was a Friday, a couple of coworkers decided to start their weekend early.

   My mistake on SunOS (with OpenWindows) was to try and clean up all the
   '.*' directories in /tmp. Obviously "rm -rf /tmp/*" missed these, so I
   was very careful and made sure I was in /tmp and then executed
   "rm -rf ./.*".

   I will never do this again. If I am in any doubt as to how a wildcard
   will expand I will echo it first.

   $ echo ./.*
   ./. ./..

  rm -rf ./. ./..

Michael Tiemann on "The Worst Job in the World":

http://www.art.net/~hopkins/Don/unix-haters/slowlaris/worst-...

>I have a friend who has to have the worst job in the world: he is a Unix system administrator. But it's worse than that, as I will soon tell. [...]

https://en.wikipedia.org/wiki/Michael_Tiemann

>Michael Tiemann is vice president of open source affairs at Red Hat, Inc., and former President of the Open Source Initiative. [...] He co-founded Cygnus Solutions in 1989. [...] Opensource.com profiled him in 2014, calling him one of "open source's great explainers."

https://news.ycombinator.com/item?id=20006186

http://www.poppyfields.net/filks/00070.html

The Day SunOS Died

    "Bye, bye, SunOS 4.1.3!
    ATT System V has replaced BSD.
    You can cling to the standards of the industry
    But only if you pay the right fee -- 
    Only if you pay the right fee . . ."

An hour later, he calls back. Hacked again. Restored again.

An hour later, he calls back. He realizes that the hacker is him! He's doing a thing, but doesn't know what he's doing wrong. So I have him email me the last thing he typed on his server, as root:

   rm -rf /home/user/path/to/thing /home/otheruser/path/to/somethingelse / home/path/to/some/other/thing/altogether

Back in 1984/5, I had a directory in my homedir called "etc" for of miscellaneous stuff. One day I thought: that's a bad name, I should remove it. I errantly typed "rm -rf /etc". Thankfully I got a "Permission Denied" error. Except, I then did the obvious override, "sudo rm -rf /etc" (1). This was on a VAX 11/780 with about 50 undergrads doing project work on. The command ran for a while and then I heard moans out in the terminal room, as the system crashed. It took us about 3 hours to restore from backup tape.

(1) I used to have to explain what "sudo" was, because this happened before we posted it to USENET, and before it was ubiquitous on systems.

Another onprem service formerly run by experts shutters its doors. In their halcyon days they wrote Exim.

You can’t even forward to gmail because the current sysadmin team have IPv6 set up without PTRs, causing gmail to 500 one in every N deliveries.

I imagines GOOG and the University of Cambridge locked in an endless battle of “don’t you know who I am”, and meanwhile the dons et al shuffle into the abyss of outlook.com penury.

What would ‘fanf2 do?

With a hard NFS mount, the mount will hang until the other machine responds.

When we had to power cycle the two servers, they would not come up as they were deadlocked waiting for each other. That was exciting.

https://www.ee.ryerson.ca/~elf/hack/recovery.html

Turns out a helpful soul had aliased "a.out" to 'echo "segmentation fault: core dumped"; touch core"

That made me learn at an early age to never have "." in my path, and to always execute things in the CWD as ./a.out (among other things).

Unix wasn't very common after leaving university and I have more PC/LAN type stories. There was one memorable moment, where I was working very late in the office and got a call. [If working late, the main line would ring the entire office and I could press the blinky light to answer.] It was one of our consultants on the west coast who somehow had a corrupt filesystem, but that machine was the one that had all the project files for the many months of consulting work that the team had been developing. [I don't recall but it may have been CVS or SVN.]

The tricky bit was that it was using OS/2 and its HPFS filesystem so the usual file utilities wouldn't work. We had a number of IBM tech books on our bookshelves (because we also did mainframe consulting) and I'd been reading about terminal streams and one about the HPFS filesystem in particular. It mentioned boot blocks, superblocks, bands, allocation bit blocks, etc.

Being young (and dumb) went with "what's the worst thing that could happen" and came up with a plan: using the DOS 'nu' (Norton Utility) copy a few choice sectors from a similar spec-looking machine and try the OS/2 equivalent of 'chkdsk /f'--the client after all was IBM known for conformity. We first had to dial-up modem transmit the 'nu' program, but then we were coping the first 18 (or so) sectors to get the boot sector, partition table, boot program or other HPFS initial sector data; then there were some sectors in the middle of the disk that served as a kind of main description table with others in bands (that we didn't bother with). Guessed the starting point and number of sectors. This was a grasping at straws Hail Mary. Rebooted the machine, let the OS/2 run its chcdsk as it detected a problem, waited a long while until it was done. Unbelieveably it all worked! There might have been a couple open files lost and some files that were recently deleted being present, but no big differences. We didn't think we needed to tell anyone. He bought me beers as promised when I came to visit.

Bonus memory: LapLink with the parallel transfer cable was the shit in those days. https://en.wikipedia.org/wiki/LapLink_cable

It's a niche situation sure, but being able to understand the system and tooling you're working with to a degree to understand what options you really have shows a great deal of discipline and curiosity, for me at least. Again, it's less about "can you figure out what to do in this specific situation" and more "can you just explain what you look at every single day in plain and simple terms? Did you ever think about it?"

It's been a surprisingly revealing question for nascent Linux admins on how they react to questioning the things they look at every single day, and how ready they are to __really__ dig into the kernel internals.

The version for those of us who came of age in the 90s is wondering what all this crap in C:\Windows\System32 is, and whether deleting it will give us enough space to install Age of Empires.

Heh ... I wonder how many years forward will people stop knowing what a 3 1/2 and a 5 1/4 disk is

Or when my "wrongly" expanded `mv` command moved all the files and directories in home into the last directory of the home directory. Which was a NTFS mount, leading to a loss of all the file permissions.

Needless to say that required a full restore from the previous night's backup since he hadn't snapshotted the VM before beginning his work. He was very angry that he had lost 2 days of work. I was very sympathetic...

C’est plus ça change…

It was glorious. Now its Debian varieties or Apples BSD abomination and nothing else.

Even windows server is getting steamrollered. All the progress is now happening elsewhere i.e. virtualisation, containers, devops.

I kinda miss the days where the iron was expensive and screwing up your Pyramid kicked 200 terminal users to the curb while you figured it out.

  Have you ever left your terminal logged in, only to find when you came
  back to it that a (supposed) friend had typed "rm -rf ~/*" and was
  hovering over the keyboard with threats along the lines of "lend me a
  fiver 'til Thursday, or I hit return"?  Undoubtedly the person in
  question would not have had the nerve to inflict such a trauma upon
  you, and was doing it in jest.  So you've probably never experienced the
  worst of such disasters....
  
  It was a quiet Wednesday afternoon.  Wednesday, 1st October, 15:15
  BST, to be precise, when Peter, an office-mate of mine, leaned away
  from his terminal and said to me, "Mario, I'm having a little trouble
  sending mail."  Knowing that msg was capable of confusing even the
  most capable of people, I sauntered over to his terminal to see what
  was wrong.  A strange error message of the form (I forget the exact
  details) "cannot access /foo/bar for userid 147" had been issued by
  msg.  My first thought was "Who's userid 147?; the sender of the
  message, the destination, or what?"  So I leant over to another
  terminal, already logged in, and typed
          grep 147 /etc/passwd
  only to receive the response
          /etc/passwd: No such file or directory.
  
  Instantly, I guessed that something was amiss.  This was confirmed
  when in response to
          ls /etc
  I got
          ls: not found.
  
  I suggested to Peter that it would be a good idea not to try anything
  for a while, and went off to find our system manager.
  
  When I arrived at his office, his door was ajar, and within ten
  seconds I realised what the problem was.  James, our manager, was
  sat down, head in hands, hands between knees, as one whose world has
  just come to an end.  Our newly-appointed system programmer, Neil, was
  beside him, gazing listlessly at the screen of his terminal.  And at
  the top of the screen I spied the following lines:
          # cd
          # rm -rf *
  
  Oh, shit, I thought.  That would just about explain it.
  
  I can't remember what happened in the succeeding minutes; my memory is
  just a blur.  I do remember trying ls (again), ps, who and maybe a few
  other commands beside, all to no avail.  The next thing I remember was
  being at my terminal again (a multi-window graphics terminal), and
  typing
          cd /
          echo \*
  I owe a debt of thanks to David Korn for making echo a built-in of his
  shell; needless to say, /bin, together with /bin/echo, had been
  deleted.  What transpired in the next few minutes was that /dev, /etc
  and /lib had also gone in their entirety; fortunately Neil had
  interrupted rm while it was somewhere down below /news, and /tmp, /usr
  and /users were all untouched.
  
  Meanwhile James had made for our tape cupboard and had retrieved what
  claimed to be a dump tape of the root filesystem, taken four weeks
  earlier.  The pressing question was, "How do we recover the contents
  of the tape?".  Not only had we lost /etc/restore, but all of the
  device entries for the tape deck had vanished.  And where does mknod
  live?  You guessed it, /etc.  How about recovery across Ethernet of
  any of this from another VAX?  Well, /bin/tar had gone, and
  thoughtfully the Berkeley people had put rcp in /bin in the 4.3
  distribution.  What's more, none of the Ether stuff wanted to know
  without /etc/hosts at least.  We found a version of cpio in
  /usr/local, but that was unlikely to do us any good without a tape
  deck.
  
  Alternatively, we could get the boot tape out and rebuild the root
  filesystem, but neither James nor Neil had done that before, and we
  weren't sure that the first thing to happen would be that the whole
  disk would be re-formatted, losing all our user files.  (We take dumps
  of the user files every Thursday; by Murphy's Law this had to happen
  on a Wednesday).  Another solution might be to borrow a disk from
  another VAX, boot off that, and tidy up later, but that would have
  entailed calling the DEC engineer out, at the very least.  We had a
  number of users in the final throes of writing up PhD theses and the
  loss of a maybe a weeks' work (not to mention the machine down time)
  was unthinkable.
  
  So, what to do?  The next idea was to write a program to make a device
  descriptor for the tape deck, but we all know where cc, as and ld
  live.  Or maybe make skeletal entries for /etc/passwd, /etc/hosts and
  so on, so that /usr/bin/ftp would work.  By sheer luck, I had a
  gnuemacs still running in one of my windows, which we could use to
  create passwd, etc., but the first step was to create a directory to
  put them in.  Of course /bin/mkdir had gone, and so had /bin/mv, so we
  couldn't rename /tmp to /etc.  However, this looked like a reasonable
  line of attack.
  
  By now we had been joined by Alasdair, our resident UNIX guru, and as
  luck would have it, someone who knows VAX assembler.  So our plan
  became this: write a program in assembler which would either rename
  /tmp to /etc, or make /etc, assemble it on another VAX, uuencode it,
  type in the uuencoded file using my gnu, uudecode it (some bright
  spark had thought to put uudecode in /usr/bin), run it, and hey
  presto, it would all be plain sailing from there.  By yet another
  miracle of good fortune, the terminal from which the damage had been
  done was still su'd to root (su is in /bin, remember?), so at least we
  stood a chance of all this working.
  
  Off we set on our merry way, and within only an hour we had managed to
  concoct the dozen or so lines of assembler to create /etc.  The
  stripped binary was only 76 bytes long, so we converted it to hex
  (slightly more readable than the output of uuencode), and typed it in
  using my editor.  If any of you ever have the same problem, here's the
  hex for future reference:
          070100002c000000000000000000000000000000000000000000000000000000
          0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f
          8800040000bc012f65746300
  
  I had a handy program around (doesn't everybody?) for converting ASCII
  hex to binary, and the output of /usr/bin/sum tallied with our
  original binary.  But hang on---how do you set execute permission
  without /bin/chmod?  A few seconds thought (which as usual, lasted a
  couple of minutes) suggested that we write the binary on top of an
  already existing binary, owned by me...problem solved.
  
  So along we trotted to the terminal with the root login, carefully
  remembered to set the umask to 0 (so that I could create files in it
  using my gnu), and ran the binary.  So now we had a /etc, writable by
  all.  From there it was but a few easy steps to creating passwd,
  hosts, services, protocols, (etc), and then ftp was willing to play
  ball.  Then we recovered the contents of /bin across the ether (it's
  amazing how much you come to miss ls after just a few, short hours),
  and selected files from /etc.  The key file was /etc/rrestore, with
  which we recovered /dev from the dump tape, and the rest is history.
  
  Now, you're asking yourself (as I am), what's the moral of this story?
  Well, for one thing, you must always remember the immortal words,
  DON'T PANIC.  Our initial reaction was to reboot the machine and try
  everything as single user, but it's unlikely it would have come up
  without /etc/init and /bin/sh.  Rational thought saved us from this
  one.
  
  The next thing to remember is that UNIX tools really can be put to
  unusual purposes.  Even without my gnuemacs, we could have survived by
  using, say, /usr/bin/grep as a substitute for /bin/cat.
  
  And the final thing is, it's amazing how much of the system you can
  delete without it falling apart completely.  Apart from the fact that
  nobody could login (/bin/login?), and most of the useful commands
  had gone, everything else seemed normal.  Of course, some things can't
  stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but
  by and large it all hangs together.
  
  I shall leave you with this question: if you were placed in the same
  situation, and had the presence of mind that always comes with
  hindsight, could you have got out of it in a simpler or easier way?
  Answers on a postage stamp to:
  
  Mario Wolczko

https://news.ycombinator.com/item?id=15802533

Pyramid's OSx version of Unix (a dual-universe Unix supporting both 4.xBSD and System V) [1] had a bug in the "passwd" program, such that if somebody edited /etc/passwd with a text editor and introduced a blank line (say at the end of the file, or anywhere), the next person who changed their password with the setuid root passwd program would cause the blank line to be replaced by "::0:0:::" (empty user name, empty password, uid 0, gid 0), which then let you get a root shell with 'su ""', and log in as root by pressing the return key to the Login: prompt. (Well it wasn't quite that simple. The email explains.)

https://en.wikipedia.org/wiki/Pyramid_Technology

Here's the email in which I reported it to the staff mailing list.

    Date: Tue, 30 Sep 86 03:53:12 EDT
    From: Don Hopkins <don@brillig.umd.edu>
    Message-Id: <8609300753.AA22574@brillig.umd.edu>
    To: chris@mimsy.umd.edu, staff@mimsy.umd.edu,
            Pete "Gymble Roulette" Cottrell <pete@mimsy.umd.edu>
    In-Reply-To: Chris Torek's message of Mon, 29 Sep 86 22:57:57 EDT
    Subject: stranger and stranger and stranger and stranger and stranger

       Date: Mon, 29 Sep 86 22:57:57 EDT
       From: Chris Torek <chris@mimsy.umd.edu>

       Gymble has been `upgraded'.

       Pyramid's new login program requires that every account have a
       password.

       The remote login system works by having special, password-less
       accounts.

       Fun.

    Pyramid's has obviously put a WHOLE lot of thought into their nifty
    security measures in the new release. 

    Is it only half installed, or what? I can't find much in the way of
    sources. /usr/src (on the ucb side of the universe at lease) is quite
    sparse. 

    On gymble, if there is a stray newline at the end of /etc/passwd, the
    next time passwd is run, a nasty little "::0:0:::" entry gets added on
    that line! [Ye Olde Standard Unix "passwd" Bug That MUST Have Been Put
    There On Purpose.] So I tacked a newline onto the end with vipw to see
    how much fun I could have with this....

    One effect is that I got a root shell by typing:

    % su ""

    But that's not nearly as bad as the effect of typing:

    % rlogin gymble -l ""

    All I typed after that was <cr>:

    you don't hasword: New passhoose one new
    word: <cr>
    se a lonNew passger password.
    word: <cr>
    se a lonNew password:ger password.
    <cr>
    Please use a longer password.
    Password: <cr>
    Retype new password: <cr>
    Connection closed

    Yes, it was quite garbled for me, too: you're not seeing things, or on
    ttyh4. I tried it several times, and it was still garbled. But I'm not
    EVEN going to complain about it being garbled, though, for three
    reasons: 1) It's the effect of a brand new Pyramid "feature", and
    being used to their software releases, it seems only trivial cosmetic,
    comparitivly.  2) I want to be able to get to sleep tonight, so I'm
    just going to pretend it didn't happen. 3) There are PLEANTY of things
    to complain about that are much much much worse. [My guess, though,
    would be that something is writing to /dev/tty one way, and something
    else isn't.]  Except for this sentence, I will also completely ignore
    the fact that it closed the connection after setting the password, in
    a generous fit of compassion for overworked programmers with
    ridiculous deadlines.

    So then there was an entry in /etc/passwd where the ::0:0::: had been:

    :7h37OHz9Ww/oY:0:0:::

    i.e., it let me insist upon a password it thought was too short by
    repeating it. (A somewhat undocumented feature of the passwd program.)
    ("That's not a bug, it's a feature!")

    Then instead of recognizing an empty string as meaning no password,
    and clearing out the field like it should, it encrypted the null
    string and stuck it there. PRETTY CHEEZY, PYRAMID!!!! That means
    grepping for entries in /etc/passwd that have null strings in the
    password field will NOT necessarily find all accounts with no
    password. 

    So just because I was enjoying myself so much, I once again did:

    % rlogin gymble -l ""

    Password: <cr>
    [ message of the day et all ]
    #

    Wham, bam, thank you man! Instead of letting me in without prompting
    for a password [like it should, according to everyone but pyramid], or
    not allowing a null password and insisting I change it [like it
    shouldn't, according to everyone but pyramid], it asked for a
    password. I hit return, and sure enough the encrypted null string
    matched what was in the passwd entry. It was quite difficult to resist
    the temptation of deleting everyone's files and trashing the root
    partition.

        -Don

    P.S.: First one to forward this to Pyramid is a turd.

http://art.net/~hopkins/Don/text/gymble-roulette.html

The postscript comment was an oblique reference to the fact that I'd previously gotten in trouble for forwarding Pete's hilarious "Gymble Roulette" email to a mailing list and somehow it found its was back to Pyramid. In my defense, he did say "Tell your friends and loved ones.")

Unix Admin. Horror Story Summary, version 1.0 (old) - https://news.ycombinator.com/item?id=721578 - July 2009 (6 comments)