Comment by phaedrus | Hacker News Reader

Comment by phaedrus | Hacker News Reader

I tried to back up a 60% full Ubuntu Linux RAID to a 4TB USB3 hard drive. I set the backup going and left it a few hours.

I was shocked to come back and find the RAID 100% full and only a few GB copied. It took me a few minutes to figure out what happened.

Maddeningly, the USB drive has a feature that it goes to sleep after 15 minutes or so, even if Ubuntu is actively using it to write files! Maybe the drivers on Windows do something that keep it awake, but on Linux it just goes to sleep in the middle of being used.

Now here's the thing, the drive DOES realize it has more to do and wake back up. But this sleep-wake causes a USB disconnect and reconnect. Which causes Ubuntu to unmount and remount the drive.

Now here's the problem, perhaps because the backup program is still making the the original mount point "busy", Ubuntu doesn't re-mount the media to the path. Instead it, gets mounted at "/media/path-1".

Since Linux uses regular folders as mount points, the old mount point at "/media/path" becomes a valid folder on the local disk. So the backup program keeps going, but now it's filling up the local disk.

I haven't found a solution for this problem that will allow me to complete a backup (or even complete a long-duration manual copy).

My personal opinion is that if the USB enclosure is that flaky then it's not worth trusting it to operate when it's time to do a restore.

You could shuck the drive and connect it directly or find a reliable USB enclosure.

Putting the disk in /etc/fstab by uuid should keep it mounted to the same directory at least, but I'll be surprised if the backup software properly handles the potential errors well. The following link has an example to remount the drive with udev when it goes away, and if the backup software isn't running as root you can mount with uid,gid set to the backup user (or chown an ext filesystem) but make the mountpoint directory 0700 owned by root to prevent the backup from writing to it while unmounted.

https://blog.backslasher.net/automatically-mounting-usb-driv...

+1 for shucking the drive. I got a few WD externals (14 TB). They exhibited the exact behaviour described by OP. Essentially the drive slept every 15 minutes or so.

Smartctl on a timer didn’t work. Touch on a timer didn’t work. I can’t recall if hdparm or some other tuning of sleep and head parking settings worked.

Taking the drive out of the enclosure worked. It got so much quieter too.

This is so weird. I have a few of the 10TB and 12TB WD EasyStores. I've been using them regularly for years and have never experienced anything like this.

Just to be clear, the drive does go to sleep when it's actually idle. Not while it's being used, though. IOW, they work as expected.

What distro/kernel version are you on? I mostly use Debian stable, which means a pretty old kernel. I wonder if it could be a regression in more recent ones?

I also use the drives with FreeBSD and have never had an issue there, either.

I'm also curious if it's aggressive USB power saving attempts with autosuspend. From my limited exploration autosuspend is usually off and e.g. requires powertop to enable (on Ubuntu LTS) or manual fiddling, but may be worth checking.

This was Arch without any special kernel, so a fairly recent one. It was a few months ago. Don’t recall the specific version.

> but I'll be surprised if the backup software properly handles the potential errors well.

YMMV but I've found rsync to handle this kind of fuckery well.

Or rclone if you need even faster parallel synchronization

I've used both of these tools with great success in the past. Infinitely scriptable in your favorite scripting language, being just simple command-line tools, too. Great for automation of backups.

Re; the issue of external drives spinning down causing a remount at a different location, I've never had that exact experience before, but I have had drives spin down more frequently than I'd like, so my answer to that was to just disable the spindown entirely. (There's articles about various ways to do that easily findable on your favorite search engine if you're curious how to do it.) Once disabled, if I find that I still want some form of spin-down for the drive, I install one of the handy available daemons that allows a much more easily configurable form of this feature, and then set a longer delay than the drive's default. (This can also be done by configuring the drive itself to a longer delay, but it's more of a pain than just editing a simple text configuration file for a daemon.)

I'm certain that this is a problem with the drive firmware or USB controller and not Linux per se.

I use several USB storage devices, enclosures on Linux everyday including using separate USB ssd caches.

Can you check with your storage device managing software on Windows on whether it has the sleep option or spin down set? I remember seeing one in the Seagate SW on mac although it has an OS level option to put HDD to sleep when not in use.

If that doesn't work you can try using hdparm or sdparm to modify the power management and spin down timer. If everything else fails then disable USB autosuspend in the kernel boot option.

Sleep is one problem.

The other problem is how the backup software reacts to the possible failure modes.

A backup tool should identify the backup destination in a reliable way and nountpoints is clearly not a reliable way to identify a backup destination

True but the parent didn't say how the backup is happening, It could have even been a rsync.

a shell script running rsync is "a backup automation software"

Alternatively, I have a Debian file server with 24Tb in internal drives and 3 x 8Tb external USB drives (WD EasyStores). The 3 USB drives are connected all the time and setup with MergeFS to appear as one massive 24Tb mount (ext4 underlying). I use rsync to do backups. The initial backup of 16Tb took many hours. About once a month I initiate a manual backup that often runs over an hour. I’ve never had this problem.

I have a Pi3 at my parents home and I rsync over ssh to a 5 TB 2.5" spinning disc attached via USB, I added the disc to fstab. The first sync took over 24 hrs for ~550 GB [0]. We both have 100/100 fiber, but USB is limiting in this case, which is nice because it kept both our connections usable during transfer. Anyway, I had 0 issues. Well, apart from the fact that I initially used vfat as the FS of the target disc so my partner or parents could just pull out the disc and use it (on Windows/Mac) when something would happen to me. But this leads to rsync issues and needs workarounds so I switched it to ext4 and told my loved ones to find a Linux nerd in the event of my demise.

[0] rsync -ahv --progress -e 'ssh -p2222' /data/0/Pictures bu.mydomain.tld:/mnt/data/backups/ --info=progress2 #(assumes cert based auth set up)

the controller in the enclosure is maybe overheating and either switching into a failsafe mode or a failure mode until it cools down again.

sustained data transfers crush those little controllers, so spend more on a better enclosure, actively cool the one you have, or just connect the bare hard drive directly to your PC if that is an option.

I've had 2 USB-3 to mSATA enclosures fail permanently after 5 minutes of sustained data transfers.

can you give the brand and model so we all avoid that?

I had related issues.

My solution since then was to work through a soft link to a directory inside the mounted drive.

Whatever the failure mode, when the storage disappears, even if the users do a “mkdir -p” (which seems to be the underlying issue you are describing)

Most linux tools have an option that keeps you from crossing filesystem boundaries

I knew that for reads how do you do that for writes?

  chattr +i /mnt/foo

This makes the mount point immutable when nothing is mounted there.

Thanks, that will save me some serious headache!

OOooo, that's a nice trick! I'm totally gonna use that in a few places in my current setup. A little shocked at myself for never havin' thought of usin' the immutable bit in that specific way. Seems so very sensible and logical now that I see it. :)

In this case he needs the opposite -- an option keeping it from not crossing filesystem boundaries.

I have a USB 2 external HDD enclosure that will drop off the bus if it is connected to a USB 3 port and under any respectable load. Only workaround I know of is to connect it to a USB port while giving the logo a stern, backwards-compatible glare.

This seems like it must be an issue with the drive you're using. I use a mix of Western Digital and Seagate USB drives and have never had a problem like this.

Huh, it's almost like treating separate volumes as part of the same file hierarchy is a bad abstraction.

Perhaps it is SMR drive. Ubuntu or linux is generally poor at sleeping USB. Ubuntu always mounts at /media/username/drivename - I am sure something is wrong with your setup.

Why do you refuse to blame Linux for this problem?

Because not everything that happens on Linux is Linux's fault, any more than every thing that goes wrong on Windows is Microsoft's fault? Sometimes it's user error, sometimes it's faulty hardware, sometimes it is the operating system. You don't just lay blame on the operating system for everything that's not working though (although I do see a lot of people do exactly that, despite it not being logical).

As an example; Many folks commenting above appear to have never experienced a problem with hard drives spinning down during a backup, or it causing a remount at a different mount point. This leads to thinking it might not be an operating system issue, but rather something strange about that specific external hard drive.

Reply on news.ycombinator.com