Comment by scarygliders | Hacker News Reader

Comment by scarygliders | Hacker News Reader

In ye olde unixe days, I learned a habit.

That habit was;

# sync

# sync

# sync

# halt

When powering down ye olde unixe maschine. It was so far back I just vaguely remember I'd read that somewhere; possibly in a ye olde unixe manual, but I can't remember exactly. I think back then it made sense, due to older spinning rust technology and there might have been some possibility of the OS not flushing any waiting cache to disk before powering down.

Even though it's probably not needed today, I still do throw a sync command in when powering down, say, my NAS with spinning rust drives, and my PCs/laptop with SSDs.

There is one thing where I 100% perform a couple of sync commands, and that's right before unmounting a USB memory stick - I usually witness linux Very Quickly Copying a file from PC to USB stick. "Aha! No, you're just caching that operation you sly dog you!" - the sync command on a CLI tells me when linux has finally actually copied the file onto the stick.

telling you to type in "sync" multiple times was the old-timers way of ensuring that the first sync completed (maybe about a second) before you turned the power off, even if you were a newbie SA.

I have become the old-timer ;)

it happens to us all!

It beats the alternative.

It's a point in time write barrier no? Everything before the sync has to get written.

But writes are still getting added after the sync. Maybe I'm wrong but it seems like sync has to not try to sync those latecomers, because otherwise it might run forever, with new writes showing up each time it waits for current writes to sync.

Assuming the above is true, it makes sense to sync multiple times. You converge towards being fully synced. The first sync might take extra long (say 30s) because there could be tons of stuff unsynced. But the next sync should only have those 30s worth of writes to sync. Let's say that takes 5s. Now the third sync only has 5s worth of writes to sync, and that takes 1s...

This is all built on some assumptions about how sync works. But it feels like sync needs to be able to terminate & these assumptions are the only thing I can imagine that makes sure sync can ever finish, without never ending changes keeping it open.

> The first sync might take extra long (say 30s) because there could be tons of stuff unsynced.

that would have to be an awful lot of data, or a very slow drive, perhaps a floppy?

your other assumptions are sensible, to an extent. but way back when, we typically had a script that sent a message to all ttys saying "system going down in 30 seconds, please log off" - in fact ALL the other systems i used back then (Dec10, VAX, IBM 4381) did this before the sync (or whatever they did).

I’ve seen extreme sync times when writing large files to an old slow USB thumb drive. Often you will see the first part of the copy run at full USB speed but then it slows to a crawl. Then the unmount (which does a sync) pays back that initial burst of speed. I’ve seen it take more than 5 minutes (a kernel warning about the slow sync appears in dmesg after 300 seconds IIRC).

Don't you have any shitty ass USB sticks that only do ~6MB/s writes? 180MB worth of dirty pages is nothing.

Only 6MB/sec? That would be lightning fast when I first started to use USB flash drives. I have used USB 1.1 drives which are rated a mind boggling ~900kB/sec sustained transfer speeds.

Good Kingstons did 5MB/sec, top of the line ones did 11MB/sec.

In any production machine, I would strive very much to avoid this circumstance, yes. Better tune your system. Either allow less dirty data in the first place (dirty_background_ratio defaults to 15% of memory... even if you have 2TB of ram) or lower your dirty_writeback_centisecs so not as many seconds of dirty writes get buffered if this happening, pretty please.

My example used more illustrative than real numbers. Still, 10s seems not totally uncommon, which is only 3x off.

You're supposed to close applications (at least ones that you cares about) then sync. Then by definition nothing important falls into the queue

Also I'm around 70% sure remounting system readonly does equivalent of sync & not allow more writes.

Apparently it is not, or at least, it wasn't, once, if you believe this research:

https://bsdimp.blogspot.com/2020/07/when-unix-learned-to-reb...

The post says BSD tried to do what I said. And it failed to actually do it right, as basically a bug. And needed to be updated to do what I said. Which took a couple decades to find out & make happen but was a bit of an issue.

Still no confirm or deny on Linux.

I was at +2 and now 0 after this post rose to the top. Am I bitter? No...

> It's a point in time write barrier no

That's the only way it makes sense. The caller controls when they start the sync, so they can make sure that all writes they care about have been issued by then.

But it's impossible to define a temporal ordering between writes and the time the sync completes. That's not a well-defined point in time. There's always a non-zero delay between the time the sync proper completes and the caller gets back control.

A coworker once asked me for shell tips. I gave them a few that came to mind, and also invited them to peruse my .bash_history, because I figured there might be other tricks hiding in there.

They later asked why I sync all the time, almost always in threes.

"Huh?" Looking through history, I see that I do. Apparently it's been ingrained into my subconscious, like a sort of stutter, and I type it sometimes out of habit when I'm deep in thought.

Old habits really do die hard.

The big one is if you dd an image to a USB stick, and then try to pull it out and boot from the stick. You can use conv=fsync, but maybe you forgot to do that when you ran dd.

I made it a habit to run `eject /dev/sda` (as root) before removing a USB stick. That flushes pending writes and makes it impossible for new writes to occur before the stick is removed and re-inserted.

Just `sync` also works but it's asynchronous, so you don't really know how long to wait until the writes are flushed to disk (especially for USB sticks which can be really slow when writing).

I still do this. There was another command which was used to park the heads, too.

Ah yes! And park the heads before opening up the server or moving it. Absolute must.

FXPARK.EXE! I'm getting flashbacks of Tandon DOS! Aaargh!

I've always been wary of USB sticks. I have some that write slowly, and the system says still writing. I have others that seem to however it works, return immediately like they're super fast, and do writes in the background.

Anymore, I just buy the sticks with light indicators. It's not done if it's not pulsing evenly...

umount will block until the writes are finished, though

Indeed it will but I just like to make /sure/ ;)

Not always.

Not always, as in it'll terminate earlier, or as in it'll continue blocking indefinitely after the writes are finished? Because I think I've experienced the second case a few times.

Not always as depending on flags you pass to the umount2 syscall.

I almost always run sync after unmount on USB sticks or SD cards. Old habits and superstitions, I guess.

Recently I saw a USB stick's activity LED stay on long after unmount. I'm not sure if it was writing or something else.

Triple sync is at the end of my backup-to-usb disk script

Possible.

The little microcontroller that does the actual write to pages of flash might actually be busy moving things around and doing other housekeeping after the disk is unmounted but while it still has power.

I wrote the firmware for a flash based system, and used powered-yet-unmounted time to save a few statis to the uC's internal flash.

I have seen that too. It generally means the drive is doing write leveling and/or other tasks. If I have time, I wait for that, too.

However, LEDs become rarer and rarer these days.

If you were an old timer, I would love to hear, how significant was Evi Nemeth's UNIX administration handbook to you?

I myself learned a ton from it, but as I understand, it used to be the bible.

> how significant was Evi Nemeth's UNIX administration handbook to you?

Never encountered it. I presume it was available in the UK, but my Unix courses in college didn't use it, and after college I started off as a CAD operator, using Tektronix SVR4 workstations with 4109 terminals running Teknicad, which is where I became quite intimate with SVR4 as I was a bit of a computer nerd ;)

there's old timers and there's ye olde thymers. The big shift was the web/.com boom. In the period before the mid 90s when high end workstations were why groups of people ran unix, sysadmins were frequently more experienced systems programmers who knew a lot about the internals and why Chesterton's various fences had been built, and they did sysadmin because nobody else could be trusted.

With the interwebs there became a massive need for many many servers, and an attendant need for many many sysadmins, and their training/experience would be purely administration, with more need to glean more information from guides.

In that same time period, use of computers shifted to more personal computers, so the demand for datacenters was from a lot of Windows9x boxes (that needed administration and didn't get any). And then the shift to WindowsNT and then AMD64, so who was "coming up" and how much they knew of systems programming kept declining.

Now we've started to adminster administration, with a lot of files in /etc that say "automatically generated, don't edit this"

Evi Nemeth's books were feeding the internet growth, not the real old time sysadmin's.

Zero. I've never read it to this day. Eric Foxley's Unix for Superusers, on the other hand ...

Foxley's book was published in 1985. By the middle 1990s there were tonnes of books being produced, none of which was really "the bible", unless you counted ones that had "bible" in the title, like the Waite Group's "Unix System V Bible". (-:

Reply on news.ycombinator.com