Unix Recovery Legend (1986)
ee.ryerson.ca
ee.ryerson.ca
But, rsync.net used to run - exclusively - on UFS2 and, circa 2009/2010 we were pushing UFS2 to the very limits in terms of disk size, number of inodes per filesystem, size of memory used to fsck, etc.[1]
Several times we came very, very close to destroying a filesystem and losing customer data.
What made all the difference, every time, was:
- don't panic
- postulate what is happening
- do something non destructive that will prove or disprove the theory
- possibly repeat
Go slow and don't take any steps until you have proven that you understand the problem.
ALSO, when working with filesystems remember that you can run a filesystem, non-destructively, in read-only mode. There are a lot of investigations that can be done on a filesystem in either explicitly read-only mode or with tools (like fsck) set to run no-write, or non-destructive.
Your first steps should always be read-only ...
[1] I asked Kirk M., in 2010, what the long term solution to our problems with UFS was and he said "migrate to ZFS".
Some years ago (but not so long ago that I shouldn't have known better...), I wiped out by mistake my home directory were I had a few days of unpushed changes (yes, second mistake) to an important project I was working on.
When I realized what I had done, I immediately remounted read-only and I managed to preserve the data; a few hours later, ext4-undelete saved the day.
It was in a VM, so rather than remount read-only, I snapshotted the VM drive, then rebooted to perform recovery, confident I could try out things non-destructively by reverting to the snapshot.
Used ext4magic to undelete, and that tool did a really great job. Some files ended up with contents that should have been in other files, so it isn't perfect. But it was a project with only a few files, so I was able to reconstruct it easily enough because all the contents were at least recovered.
Did something similar for a customer a few years ago who had wiped a significant directory on a live system. That was on a RAID-1 (mirrored).
Rather than remounting read-only, to allow production to continue I detached one of the mirrored drives to preserve it for safekeeping, while allowing operations to continue using the other drive.
https://marcan.st/2015/10/rescuing-a-broken-ntfs-filesystem/
I managed to figure out what the problem was and undo the corruption without data loss, but it was quite a deep dive...
It was linked to in an LWN article's comment section, titled "If you think you're really good, read this.", or something. The story proved very effective at humbling its reader for me.
That doesn't change the fact that I, too, once felt like the uncrowned teenage king of emergency system rescue operations back in 2002 or so, right in my parents' basement. I had started recursively deleting / by accident, and ended up with most directories, /bin and /sbin included, already turned to dust, and most of my as of then proudest achievement - a Gentoo Linux stage 1 installation - gone.
I dug myself (and the standing remains of my /home) out of that particular hole by virtue of still having an instance of a browser (I think firefox) running. Mildly panicking, I still left everything untouched ... and after a few hours of thinking hard about what to do, I arrived at the following: I set up a webserver on my mom's windows PC with am embedded PHP interpreter, and hacked up a script that enabled file uploads via HTTP over the LAN. After using that from within my still-running browser (fortunately, it could still list the contents of my home directory) to evac the most precious artifacts my ~ had in store, I rebooted my machine into the lightless nothingness of not having a kernel image in /boot any more. Another few hours later, I had Gentoo reinstalled, and copied the rescued files over to my own machine again - and felt like I had levelled up a little bit.
If I were him, the best I could imagine was doing documentation reading and ASM writing on a separate computer with working toolchains, testing it, and finally entering some machine code in hex to the broken machine by hand. But he literally wrote an ASM program and constructed an ELF on a totally broken machine itself on the fly, with no documentation, no toolchain. Absolutely wizard.
For example, it was how the syscall number was found without grep...
> The next step was getting the syscall numbers. grep? We don't need no stinkin' grep.
# while read i; do case $i in *__NR_link*) echo $i;; esac; done
</usr/include/asm/unistd.h
#define __NR_link 9
# while read i; do case $i in *__NR_exit*) echo $i;; esac; done
</usr/include/asm/unistd.h
#define __NR_exit 1
#define __NR__exit __NR_exitWhen I realized my mistake, my heart sunk. I got cold sweats. I was afraid I'd get fired. To this day I'm so glad my team leader was still at the office at the time. I told him about the blunder, and he asked me about the details. He sat down next to me, and we started working together, exchanging information and getting replacements back up. There was no CloudFormation, so everything was manual.
He never accused me in the postmortem, and I still feel so grateful for it. I learned from my mistakes and took action to prevent it from happening again.
Listen when you break something tell us. Explain what you did and we can get it fixed. You will make mistakes, we all still make mistakes.
That is what i always do and expect my coworkers to do. Yeah we talk a little smack afterwards (only jokingly), but were able to resolve most self inflicted wounds quickly. Nowadays what scares me more is a coworker who hides mistakes rather than one who confesses to them.
I ended up using gdb (or maybe lldb I don't recall) to load a shared object that did a seek()/open()/copy(). Totally hacky but it worked!
However this story took place on DEC Unix about 10 years earlier. So /proc was not available.
Windows lets you delete files which are open all you want, and Windows also lets the application which opens the file prevent that. A lot of things like this are blamed on Windows when it's actually Windows providing options to the application developer, and the application developer choosing to take advantage of options like that when they don't need to.
see: https://docs.microsoft.com/en-us/windows/win32/api/fileapi/n... and note in the left side nav that reading, writing, and locking are all separate operations.
(This is a common misconception that I wanted to clear up, and isn't super relevant to the topic at hand.)
Windows programs also often only support 'CDs' (ISO9660 / UDF), FAT, NTFS, and maybe SMB/CIFS (Samba / CIFS) network shares. Since Windows lacks (or at least historically lacked) a general VFS they could target. That's why any actual VFS that ever gets made, due to all the legacy software, is either going to directly support one of those interfaces, or have a compatibility mode which does.
I would have thought that even a separation between OS and User applications could allow them to implement features like a VFS, while still maintaining compatibility. Are they not part of the way there with Hyper-V + WSL?
You would not believe some of the business critical software out in the field. They rely on very obscure bugs being present in order to function, and they sometimes can't be fixed because the source code is missing.
Apple has chosen the opposite path, which is to give the middle finger to long-term backwards compatibility, and advance their operating systems technically.
You can't run an OSX game from 2002 on the Macs of today, but you almost always can run a Windows game from 2002 on the Windows 10 PCs of today.
Sometimes you can simply log out and log back in, and the requirement to reboot is nowhere nearly as frequent as it used to be, believe it or not, but yes you still have to reboot for kernel patches and core service updates, and there are a lot of those things these days.
I have done something similar to recover data that was still available in a process but no longer in any file - an overwritten configuration file.
Be careful though - one mistake and you can crash the process, losing the file!
He was so wrecked about it and we, the more senior guys started helping him and comforting him that "yeah, this kind of blunders can happen to anyone, no worries. Let's figure this out together.".
The other guys figured how to restore the data into correct setup on our end, and I found some unimplemented badly documented "undo invalidation" endpoint from the CC operator with which we could fix the stuff on the operators end.
So we spent nice and fun evening together figuring out the problem as a team, but damn the guy was so torn up about his small and completely understandable fumble that nobody else needed to critique him about it.
The funny thing is that with this kind of blunders one learns a lot more and a lot faster than usually. People need to make mistakes. It's usually the best way to learn.
• Thomas John Watson Sr., IBM
There's a Norwegian saying that damage makes you knowledgeable - but not rich.
Learning from others mistakes are very often cheaper and/or less painful.
Hilsen fra Sverige (^_^)
(though I'd translate "klok" with "wise")
van schade wordt men wijs, maar zelden rijk
- Door schade en schande wordt men wijs (dutch, "through harm and shame, one begets wisdom")
- Durch Schaden wird man klug (german, "through harm one gets smart")
- Dommage rend sage (french, "stupidity breeds wisdom")
- Quae nocent, saepe docent (latin, "what hurts often teaches")
For anyone still following I think "wise" maps to the Norwegian "vis".
BTW: I think old proverbs are really interesting - both local ones as well as international ones. Sooner or later someone should try to create a complete encyclopedia of proverbs :-)
The jokes circling the office for the next few weeks were all variations on "Remember alyandon - r doesn't mean 'revise'". :(
I've never understood why we accept such a terrible interface as some intractable reality.
It was a drinking game where we would set up a Windows 98 machine and take turns randomly deleting a file found in the system32 folder, then rebooting.
If the computer fails to boot sufficiently to allow the game to continue, you lose and finish your drink (and have to reinstall windows).
It was pretty fun because windows was quite good and whining and complaining about countless problems but still being functional.
The most reasonable solution seems to use something like Jenkins--but I don't think anyone is happy with that.
At one job they wrapped crontab and used RCS to write to a network location. Now you can grep a single folder, its now version controlled, and you can wire any re-image script to restore crontabs. I've pitched this approach at subsequent jobs, but was never in a seniority role to get it rolled out.
Jenkins is a very capable app, but it's not "light" or approachable. Introducing it to new groups of people for an off-label purpose is a tough sell. If you're already heavily using Jenkins and that same group of people need it for periodic task scheduling it's an obvious choice.
If node.js: setInterval for short intervals, Bree for more complicated tasks [1]
If C: time_t
If Java: java.util.Timer, probably something 3rd party that better is wrapped around this
Anyways, the bigger point is that you should be expressing these interval settings in code, not system configurations. Everyone checks in source code but nobody checks in the crontab. It's better if you rely on the application to do these things rather than the system, because the system could always change beneath your feet (i.e. management says we're using Google Cloud now! Good luck porting a 200 line crontab to GCP) but you are in total control of your application's source code.
https://manpages.debian.org/buster/systemd-cron/systemd-cron...
It creates transient unit files based off of a standard crontab on boot and after any file modifications. If you want to extend one of your cron jobs with something that you can't express in a crontab you can just mv it from /run to /etc and modify it however you'd like.
Extra bonus points if it yoinks things the other way and lets you edit them as a nice crontab file that makes sense before re-integrating it in to the nightmare syntax.
My poor cowoker still feels embarrassed about it, even though it's been some years since.
So I came in one afternoon shift and my colleague said "your script destroyed two Production servers last night". These servers were critical for monitoring Police radio GPS coordinates.
"Impossible! It should only work on directories we configured it for!"
I sat there looking at horror at the debug logs, and sure enough, it read the empty child element, changed directory into "", looking for files with "" name pattern, which for Powershell resulted in C:\Windows\system32 and all files. It proceeded to zip and remove "old" files. On a reboot, they did not come back up.
To add insult to injury, when they rebuilt those servers manually, the same script with the commented out config was deployed, destroying the servers again. They realised that the script was the issue and decided not to deploy that the second time.
So I learned a lot about config validation and defensive coding in the months following the incident.
2015 https://news.ycombinator.com/item?id=10160417
The decision making here does not make much sense to a youngster (36) like me.
Why couldn't they just mount their half deleted drive on an other machine and restore it from there?
I have the feeling all these other hundreds of seemingly easier solutions were out of the realm of possibilities back then, but I am unsure of what were the actual limitations / constraints of that era.
Later I run into the same problem (again with npm), but this time there was not enough of unnecessary files to delete to be able run the balance (package manager cache was already purged). Normally this can only be solved by adding new partition to the btrfs filesystem and I had no unformatted or unused partiton on disk left. Instead I wanted to get rid of the btrfs that failed me twice and rsynced whole btrfs content into a loop-mounted ext4 disk image on a NTFS drive and then after unmounting I dd-ed the disk image over the btrfs. It booted! Had not any more filesystem trouble ever since.
What are the responsibilities of the OS's Kernel if/when all files are deleted, and all processes are similarly deleted?
In other words, the worst-case of all worst-case scenarios.
Under that scenario, philosophically, OS-design-wise, what are the responsibilties of the kernel?
Well, we might start by designing our future OS with a tiny shell inside of the kernel, for emergency use only.
Also, we might consider using a file system that doesn't actually delete files when they're deleted, but rather flags the disk space as "empty" -- for future recycling.
Between these two things, if they were implemented, we'd get a lot of leverage to resurrect a dead system, but there's more to this than that.
You see, the above scenario should become an active unit test (tested at each recompilation) of that future OS.
Also -- we want to know the fewest number of steps necessary (given the above scenario) to fix everything back to normal.
In other words, a very interesting topic for future OS designers to think about!
Practically speaking, none. The responsibility of the kernel (in Linux terms, but I think it's the same across all Unixen) is to start the init system (PID 1), and that's where it ends. Should PID 1 ever terminate, the kernel is in an undefined state. The VFS semantics ensure that even when the init binary is removed from the filesystem, it is not actually unlinked until its reference count is zero -- and since the kernel holds a reference to it, the kernel and init are "safe" from harm through deletion.
Well, we might start by designing our future OS with a tiny shell inside of the kernel, for emergency use only.
While not in-kernel, systemd is ahead of you, as it offers a shutdown-time ramfs to use as root filesystem so the real root filesystem can be safely unmounted. Previously, Linux didn't really unmount the root filesystem (it couldn't, because the kernel still had an open handle to the init binary), but mounted it read-only before shutdown.
we might consider using a file system that doesn't actually delete files when they're deleted, but rather flags the disk space as "empty"
This is already the case. "removing" a file in Unix only removes its name from the directory listing, the inode isn't reused until it's been cleared. And even when it's been cleared, only the inode entry is zeroed -- not the data contents. That's why tools like "shred" exist -- to make sure a file's contents are removed along with the file.
a very interesting topic for future OS designers to think about
I think transactional filesystems, snapshots, and log filesystems already do most of what you're asking for.
So, I appreciate that!
Also, I have no question that you are a smart person.
But, that being said, I'm going to play Devil's Advocate a little bit here, if that's OK with you, as I don't think that people learn anything from agreeing with one another; but rather by the friendly and polite challenging of one another!
OK, so with that in mind, here we go:
>"Should PID 1 ever terminate, the kernel is in an undefined state."
Undefined by WHO? If there's nobody that takes the responsibility for defining something, then there is no rule, law, custom, or precedent there, and all of those things may now be implemented as seen fit by one or more implementing parties.
If that's the case -- then why can't it be implemented differently, why can't that be changed in the future?
>While not in-kernel, systemd is ahead of you
That might be true -- but I am NOT a member of the systemd religion! (You know, like you have Christians, Muslims, Buddhists, Hindus, Jews, and a smattering of other smaller groups in that bunch, like Wiccans, Scientologists, Deep Staters, and people that follow Joel Osteen! <g> Well, you have all of those, AND you have the people that believe in systemd! Of which I am not one of! <g>)
On a serious note however, systemd, while it might create a whole lot of "conveniences" -- creates equal-and-oppositely many drawbacks in terms of code complexity and the ability for someone even moderately technical to understand what's really going on on their system.
But, all of that is a "religious debate" -- of which we've already seen too many of on the Net...
>This is already the case. "removing" a file in Unix only removes...
Yes, this is well known... However, let's say we get a future OS student up to speed by building their own operating sysem... if they're building their own OS, then their file system will probably not do this. It a separate consideration and it adds to the complexity of the file system.
On the one hand, yes, you're quite right, such a student could just take pre-made source code for a filesystem and link it to their OS, but if they did that, then odds are they'd learn nothing of value...
>I think transactional filesystems, snapshots, and log filesystems already do most of what you're asking for.
I don't question this.
But, if you took the world's most technologically complex airliner, whether that's a Boeing 747, an Airbus A-380, or what-have-you... most people would rather fly on one of these things than even attempt to build their own plane, even if their own plane would be simple and crude by comparison, even if their own plane would be many future iterations removed from the current state-of-the-art...
Well, even if someone attempted something like this and failed(!) (e.g., they tried to build the equivalent of a Wright Flyer -- the first heavier-than-air plane built by the Wright Brothers) -- well, that is far, far superior educationally (in my opinion!) to someone who uses the latest and greatest technological marvel of the world -- and, yet, doesn't completely understand it...
You might say that that's my religion...<g>
It's the same thing with Operating Systems...
>I think transactional filesystems, snapshots, and log filesystems already do most of what you're asking for.
I completely agree that they might...
But you know, we might agree that Windows 10 does most of what 99% of computer users want -- so why use any Unix derived variant?
I completely agree that you're right; however, Windows 10 probably does most of what you're looking for OS-wise... yet you seem to be more inclined to use a Unix derrived OS...
Why?
?
Unfortunately unlike the cool cucumbers in this story, I hastily rebooted, and it took a fair while for various reasons to get hold of a boot disk and get it repaired and running again. I love Gentoo but it does take a lot of patience generally, and this kind of thing seems to happen fairly continuously (which teaches you a lot but requires that you want to learn and solve the problems yourself).
That doesn't sound so bad right, there's nothing important just in the root directory - except in those days that's where the system kept the kernel binary! No problem, I'd learned how to do network file transfers the day before so I just copied the kernel across from another ELC and rebooted.
Phew!
I've not used Unix on an early VAX before, but every other OS I've used, including DOS, Windows, Linux, and macOS has the option to use a "recovery mode" shell if you boot from the OS installation media.
The second file of the first tape contained a miniroot filesystem which, at the very least, certainly would have included working copies of of the two utilities mknod(8) and restore(8) required to restore from his dump(8) tapes.
As with DOS, Windows, Linux, etc., 4.3 BSD could only mount filesystems stored on random-access block devices (so not sequential-access magtape).
Therefore, one of the first steps in "booting from distribution media" was to copy this miniroot filesystem to the swap partition of a formatted and labeled disk, at which point it could be mounted as the root filesystem (with swapping to this partition disabled, obviously).
In the (likely) case that the existing disk layout contained a swap partition of sufficient size on a device supported by the boot loaders and kernel supplied by the distribution, this would have indeed been a reasonable road to recovery, and, assuming the existing disk labels remained valid, the required swap space was available, and the dump(8) tapes were good, shouldn't have required anything beyond:
1. Copying the miniroot from tape to a preexisting swap partition using the standalone copy program included with the distribution media (typically on a console tape or diskette as appropriate for the VAX model in question).
2. Booting vmunix from the miniroot in the swap partition, taking care to RTFM and include the special syntax required to prevent swapping to the partition containing the miniroot.
3. Recreating required special files with mknod(8) (possibly using the /dev/MAKEDEV script from the miniroot).
4. Using restore(8) to restore from dump(8) tapes as usual.
When I tried to login again after they rebooted everything, I found that my account had been banned. I got a NOC admin on the phone and apologized for my mistake, but also asked him why they did not enforce a maximum number of processes per user (via ulimit). Apparently they had never thought of doing that. He thanked me and my account was restored immediately.