Hacker News Reader

Comment by nly | Hacker News Reader

It's a lot easier just to reboot and let fsck patch up your filesystem... flushing buffers isn't going to help you get back the last paragraph of your PhD thesis if Libre Office hasn't called write().

In any case, in ~8 years of using Linux, I don't think I've had a freeze that was the kernel and not just X.

View on HN

One easy way to get kernel lockups is to use fglrx.

But agreed on 'just reboot': you should be using a resilient filesystem anyways, and MagicSysRq only works when it's enabked beforehand, and only for some kinds of lockups, and if the data is in the frozen application, syncing the disks isn't going to help.

This is the most elaborate scheme to raise one's relative global uptime I have ever seen.

>One easy way to get kernel lockups is to use fglrx.

Yup. Changed my GPU to nvidia, no more core lockups.

For me, I kept the hardware, and switched to the open-source driver.

And what if it's not your PhD thesis in Libre Office but a busy database server which can easily get corrupted if you don't flush.

> I don't think I've had a freeze that was the kernel and not just X.

Device drivers sometimes have bugs, especially when the device is not working properly (I had system freezes when there was a misbehaving capacitor on the graphics card).

> And what if it's not your PhD thesis in Libre Office but a busy database server which can easily get corrupted if you don't flush.

Wouldn't that make it an unsafe database server?

Yes, I agree, but just several weeks ago I had to recover a MySQL InnoDB database after a power problem. MySQL is very popular, and it turns out it is an unsafe database server according to your definition. Well maybe it is.

Edit:

Besides, hard drives have write caches, and they can report a successful write operation to the OS when the data is still in its cache physically.

I don't believe MySQL is ACID compliant. You may have just ran into one of the reasons how.

MySQL is only ACID compliant under a very specific set of configuration parameters. This makes it even more dangerous than an RDBMS with binary acidity.

Yep, MySQL is an unsafe database. The fact that it's popular just goes to show how powerful marketing is on th emind of most people.

Hard drivers are supposed to flush their caches before they report the end of a flush operation to the OS (some flush into flash, but they flush). If your does not, it's defective. Go ahead and make use of the warranty.

Re HD lying, this is rare except with dubious cheap flash sticks. The caching command set is well defined and operating systems know how to issue the relevant SCSI/SATA commands. It's critical for correct functioning of journaling filesystems such as NTFS and Ext4.

> And what if it's not your PhD thesis in Libre Office but a busy database server which can easily get corrupted if you don't flush.

No properly configured and working ACID compliant RDBMS should lose any data when the server is reset or stopped. If it does, then it is either a problem with the hardware, OS, configuration or the RDBMS itself. The application must also be able to handle the DB disappearing, though. Sadly this is often not the case.

You mean lose data after it's been committed. I can send something to the database just as it's dying and it's been lost.

SQL commit is not allowed to finish before the data is safely on disk (that's the D in ACID).

That's assuming that the RDBMS can tell. The disks may lie:|

Thus the 's' in reisub?

The disk may lie. The 's' will get the OS to send everything out to the disk. Actually writing it to the platter (or flash part, or whatever) is at the disk's discretion.

It's at the disk's discretion as far as the laws of physics are concerned, but this would a severely broken disk prone to losing data and if it was a major server disk vendor, the vendor would take a pretty serious hit to its reputation.

Or the CPU or the memory or the OS...

You should strive not to use such disks in your server. A machine reset won't power off the disks, though.

Yes, but my point was that I could send data to an ACID compliant server, and kill it before the commit happened and data will be lost. Just trying to point out to the parent poster that sending is not enough, you need to wait for the commiting.

"sync; sync; sync; shutdown -r now"

I remember some sort of magic invocation like that years ago for our medical "device" product we had to manage hundreds of remote node instances of. Something along those lines. I don't remember why the developer (Gabe) came up with sync three times being magic number. Maybe three times was just paranoia. =)

"sync, sync, sync your file systems gently down the shutdown -r..".

Back in the bad old days of pre-UNIX Macs, it was a common troubleshooting step to reset your PRAM. This is battery backed up RAM that holds some basic settings, and if it got corrupted somehow it could cause weird problems. You'd reboot while holding down command, option, P, and R, then wait for the boot chime to sound a second time indicating that it had been reset, then release the keys and boot normally.

Somehow this advice got mutated so that you'd keep holding the keys until you heard two boot chimes (thus resetting the stuff twice). And then it started to grow. Three was common. Some people would advise more. I'm pretty sure that doing it more than once never helped anything, but there we are.

(The cmd-opt-P-R sequence still works on modern Macs and I actually used it to resurrect a machine that wouldn't start up just a month ago, but it's far less frequently needed now.)

It's the same with the battery stats resetting and the Dalvik cache wiping these days in Android land. You do it three+ times.

Or the "Repair permissions" thing in OS X. You do it several times as well.

It's like whenever there's this one-step fix thing that a system utility does, the Common Man will interpret it as needing to repeat 3+ times in order for it to be effective.

I believe the thinking is because sync isn't instant, especially on older slower hard drives, having to type it again give it time to actually complete.

sync will block until it completes.

Back in the day (old Unix), the sync call would return right away, and the kernel would sync in the background. Unless there was a current background sync happening -- then sync would block until the first one finished, which is why you would have two sync's in a row. The third sync was thrown in just for luck.

One for the Father, one for the Son, and one for the Holy Ghost.

Somewhere in the dim and distant past I was told (or read) to use:

"sync; sync; sync; halt"

Presumably so you had thinking time before automatically restarting a possibly sick system.

This is likely a sun thing. Halt on (SPARC) Solaris does not shutdown the OS. It issues a reset command to the firmware. The halt command on Solaris is roughly equivalent to pressing the reset button on a PC.

When your Sun is particularly hosed we used sync;sync;sync; halt to reset and (hopefully) not lose any data (sync forces OS write buffers to purge)

Possibly related: http://brad.livejournal.com/2116715.html

>Run it and be amazed how much your disks/raid/OS lie. ("lie" = an fsync doesn't work)

>It seems everything from PATA consumer disks to high-end server-class SCSI disks lie like crazy. Yes, that includes SATA there in the middle. I'll discuss fixing your storage components in a second.

I picked up the "sync three times" thing from an AIX kernel developer, who did it from before AIX had a "shutdown". (Yes, he would do "sync, sync, sync, power-off".) My theory was that using it three times gave the system time to actually sync the data.

I wouldn't advise other people doing this as the filesystem can get seriously broken under certain circumstances.

For example, on a Linux laptop with the hard-drive encrypted with dm-crypt, I simply lost access to my drive due to repeated hard reboots. I don't know if I could have recovered my data from it or not, but after repeated attempts of googling for the error message and following advice I simply gave up and later reinstalled everything from scratch (it's a good thing I constantly make backups ;-)).

On Linux REISUB has been my friend.

when doing kernel development on embedded systems or on a real host (not within a virtual machine), it's sometimes useful to get system information when it crashes. the cool thing is that it's also possible to send the sysrq through a console serial port by first sending a break command.

anyway, someone already mentioned a few mnemonics, I learned one, a quick googling lead me here:

http://fosswire.com/post/2007/09/fix-a-frozen-system-with-th...

I had many because of a not 100% stable hard drive + mdadm. Kernel panics during prolonged periods of high HDD load.

Reply on news.ycombinator.com