Comment by Florin_Andrei | Hacker News Reader

Comment by Florin_Andrei | Hacker News Reader

Parent Full thread

I've actually used Ext2 in production, back in the '90s. That's all we had back then. A bit later I've also played a lot with XFS, both on Linux and Irix (I've worked at SGI for a while in the '00s).

Back in ye olden days, yanking the power cord basically guaranteed a corrupt Ext2 FS and a visit from fsck. However, in the vast majority of cases, fsck would actually do the job. You had to peruse lost+found and recover the lost souls out of it, but that was usually the extent of it. I did 'fsck -y' many, many times - in most cases with good results.

XFS on PC was worse. It tended to be stupendously fast when dealing with lots of I/O to/from very large files (video editing, running VMs), but a power failure on PC was a lot worse than with Ext2. There was a good chance you would lose the whole volume.

On Irix / SGI hardware it was a very different story. Those suckers were quite reliable. Heavy as hell, too.

---

One habit from back then that's very hard to shake off is to run sync before reboot, often as "sync; reboot" - you know, just in case something gets stuck on the way down and you have to hit the reset button. A more extreme example would be to manually stop all services except sshd, then do "sync; reboot".

It's completely unjustified today, and yet I still do "sync; reboot" occasionally. It's baked into the muscle memory of my fingers after sleepless nights caused by losing stuff due a flaky driver that froze the system on reboot.

If you're concerned about flushing writes consider `mount -oremount,ro` -- this guarantees writes are flushed and is the only truly important step of a system shutdown anyway WRT filesystem integrity. Once filesystems are mounted read-only you can safely power off the machine.

Filesystems can be remounted read-only via the serial console as well, with a break-u. Useful even when userland is otherwise inaccessible such as in the event of a fork bomb.

Nice to learn this - but how does one send a "break-u" e.g. from inside minicom, or in KVM virtual machines, via virsh console?

I usually use `cu`, but the docs for minicom tell me it's control-A followed by F to send a break signal. Then just hit "u." There are a number of single characters following a break with actions to them, the full doc on this kernel feature is here: https://www.kernel.org/doc/Documentation/admin-guide/sysrq.r...

I'm not terribly familiar with KVM but the VM console tools probably have some way to generate a break signal. Check the docs for how to "send a break."

Oh, and don't forget to enable this feature via sysctl -- per the above linux doc.

"Once filesystems are mounted read-only you can safely power off the machine."

In Ubuntu/Kubuntu there was once a key combination to forcefully unmount all file systems. THey seemed to have discontinued this? No?

You are looking for the Magic SysRq Key: https://en.wikipedia.org/wiki/Magic_SysRq_key

- Alt+SysRq+u remounts all filesystems readonly.

Looks like some of the Magic SysRq Keys are disabled by default now in new Ubuntus.

https://askubuntu.com/questions/11002/alt-sysrq-reisub-doesn...

Yup. You need to set kernel.sysrq=1 , probably in /etc/sysctl.conf

And, of course, in the early 00's those habits got ingrained into init scripts. Every distro would automatically run a fsck after a forced shutdown, and sync after the heaviest deamons were stopped.

Somehow, I remember filesystems to be more reliable by them, and the time needed for the fsck being the biggest issue. Then we had a Red Hat person demonstrate a forced reboot in a computer running ReiserFS in my university, and I think in the next month no computing student was suing Ext2 anymore.

> the time needed for the fsck being the biggest issue

Right, that was always a big problem.

You mean "using"? I always was happy with ReiserFS. What a pity.

In the end EXT3 became much better anyway. It just took some time.

I still do: sync; sync; shutdown now -r

Before you think I'm too old-fashioned, it's at least the shutdown command and not "init 6"

I'm old enough to still retain the triple-sync habit: sync; sync; sync; reboot. A habit picked up on SVR3.

I worked with a guy who did that; I wonder what the original thing was that required a triple sync? It was never needed on Linux as far as I know.

To my understanding, the idea was to actually type 'sync' thrice, pressing return in between. The time it took to type it in again supposedly allowed previous sync to finish.

There's more about it in this old usenet thread: https://groups.google.com/forum/#!topic/alt.folklore.compute...

I'm one of those old-timers who still has 'sync;sync;sync' in muscle memory, and I also remember why we did it way back in the day - in my case, I learned it during days of development on MIPS Risc/OS, and what three rapid syncs in succession did, was, tell the TAPE driver to rewind the tape.

Mass-storage back then was often done with tape decks, which were a lot cheaper than hard disks, and the triple-sync trick was a common firmware trick that tape-drive vendors used to get a cheap 'rewind command' out there for their users, without requiring platform-specific bins for the job.

Ah, that's interesting to know. Was the development system also RISC/OS, or did some UNIX variant do this as well? I thought in UNIX, tape was always used with utilities such as tar, not mounted as a part of filesystem. That should of course be possible, for the patient at least :).

It was definitely possible, and some of us remember the joys of spooling data from tape before we could use it. :)

And if I recall, this was a built-in of the tape-drive itself, which could be used on multiple different systems, and so the triple-sync being used to rewind wasn't Risc/OS specific; I remember using the same tape-deck later on Linux and Irix machines, same ol' sync command worked every time.

I can still hear the whirr in my mind ..

Same here. I actually just did this today doing some work on Linux VMs where it's totally not needed. Old habits die hard.

In some ways, you're saving yourself from not doing it that one time you find yourself on (very) old hardware that still needs it. Some may laugh, but I'm sure there're more than a few legacy installs out there - just waiting to die because somebody forgot an old trick.

Always relevant in this context, a parable from Theodore Ts'o about XFS & FS reliability: http://zork.net/~nick/mail/why-reiserfs-is-teh-sukc

ReiserFS was so fast when you had LOTS of tiny files, like with a caching proxy.

Eventually the Ext family caught up with it.

Yeah, I ran ext2 in production a ton. I actually don't remember ever losing and subsequently finding anything in lost+found; in fact, almost every single one of my fsck or fsck -y were successful.... EXCEPT

One time on my personal machine, someone talk bombed me once, and I somehow had to not only recover the superblock from about the 8th backup location (yeah, the first 7 or so failed), but I also had to manually recover some files based on the inode data. But it worked!

I always had the same issues with ext2. Dozens and dozens of errors, fsck -y, kernel panics, and then giving up. It wasn't until ext4 that I started to see some serious reliability.

Especially given how often I have to hard power-off machines because they just freeze, or because shutdown -h now doesn't work, or because they keyboard input doesn't work ...

>" It tended to be stupendously fast when dealing with lots of I/O to/from very large files"

Can you or someone say what was it in the design that allowed XFS to have that performance with large files? Was this mostly extent based allocation or something else?

not uncommon in embedded world.lots products are still using ext2. "sync" is definitely required before rebooting ext2 fs. We all know :D

Reply on news.ycombinator.com