Debian-installer, mdadm configuration and the Bad Blocks Controversy

Hi, article author here. That's interesting! I'd possibly never noticed this because my serial console session has always been itself inside screen (now tmux with screen bindings), so doing that would only ever have sent me to my own next window!

I'll have to try it again next time I am in there, with a double escape.

Hi, article author here. I just tried that out on an Ubuntu 18.04 machine but it didn't work:

  $ sudo mdadm --fail /dev/md0 /dev/sdb1 --remove /dev/sdb1 --re-add /dev/sdb1 --update=no-bbl                                        
  mdadm: set /dev/sdb1 faulty in /dev/md0   
  mdadm: hot removed /dev/sdb1 from /dev/md0
  mdadm: --re-add for /dev/sdb1 to /dev/md0 is not possible
  $ sudo mdadm --add /dev/md0 /dev/sdb1 --update=no-bbl
  mdadm: --update in Manage mode only allowed with --re-add.
  $ sudo mdadm --add /dev/md0 /dev/sdb1
  $ sudo mdadm --examine-badblocks /dev/sdb1
  Bad-blocks list is empty in /dev/sdb1

Any ideas why? md0 is a simple RAID-1 metadata version 1.2 array.

linux.die.net man pages tend to be badly outdated. And you can't easily tell this is the case, because because they strip version information.

For example, this one doesn't document the -B option, which was added in 2011(!):

https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/commit/...

Better man page link:

https://man7.org/linux/man-pages/man8/badblocks.8.html

> However in my (limited) experience once you get badblocks the disk detoriates often.

This used to be my experience as well. However I've had a couple of WD Reds that developed a few (~20) bad sectors, one two years ago the other three. Both have been stable since, and I run periodic scrubs on them.

When the first one happened I had planned to swap it out, but was a bit low on cash right then so waited a bit. Then I saw it didn't deteriorate further so I go curious and decided to wait. Still waiting.

Feels like handling OOM; an abstraction breakdown you think you can cleverly work around but is actually not easily handled at all.

Generally, a small handful of bad blocks (and they come in groups due to the physical format and sector size emulation) aren't an indicator of imminent trouble. If they can be written to properly, then that could just be random data corruption caused by e.g. an unlucky hard shutdown, vibration, or other random thing. If they can't be written to, it could just be very minor drive damage or a weak area of the platter (and in this case, the drive will remap them and increase the bad block count in the SMART data)

Now if you start getting into the hundreds, then yeah, the drive is dying.

You should see what some of the other alternatives do. I just decommissioned a server and pulled the drives out of it. One of them had SMART data indicating that it reached the "drive failing, replace now" stage six months prior. The hardware RAID controller in the server had been content to give no indication of this at all.

Had another one where the drives were in two external containers and one of them lost power for a moment. The hardware RAID controller thereby marked that half of the drives as failed. This was too many failures to run the array, but it also had no mechanism to assemble an array using the perfectly operational drives it had marked as failed, so that was a restore from backup.

Meanwhile mdadm now supports --write-journal which moots the only real advantage hardware RAID ever held.

> For the hobbyist I don;'t see any reason why you'd really want to have an mdadm array

The "don't even dare to use ZFS without dedicating 8-16 GB of memory to it" warnings all over the ZFS documentation online has served as a very, very good reason for me to not use ZFS for anything at all.

I'll stick with tooling that lets my homelab actually have enough RAM to run the applications I want to run.

The reason you don't use mdadm is because it keeps track of bad blocks by default, although that can be disabled, but when using the debian installer to set up arrays it sets up the config file in /tmp then blows away any changes made when dropping out of the installer to modify the temp conf on return to the installer, making it impossible to set up a md array that doesn't track bad blocks when using a serial console (although if you aren't, you can open up another terminal to do it) for the install?

That's precisely the reason you don't use mdadm?

I think this has nothing to do with MD vs ZFS.

The key issue is that I think it's crazy to tinker around with 'bad blocks' in the first place.

A hard drive with bad blocks that it cannot conceal from the end-user is broken and not to be trusted. Replace it.

Because as a hobbyist, the fact that I can expand my RAID6 array one drive at a time is, for me, mdadm's killer app.

It's on new devices in an installer context for a RAID-1 as the text you quote describes, so yes I do know what I'm doing. But fair enough, it might give other people bad ideas, so I'll remove it from the examples.

When you're booted in the installer environment, /etc is writable. Some of the sample code in the article shows writing a file in /etc, and the file write works fine, mdadm just doesn't read the file.

I believe the reason is that mdadm in the installer exists to create arrays for the installed system, so the installer's RAID configuration needs to supply a configuration file, and for some reason that's handled by having the installer's RAID configuration write it to /tmp/mdadm.conf and having mdadm read it from there. (Rather than, say, writing it to a temporary file and directing mdadm to read that temporary file.)

> > the mdadm binary in d-i is compiled to have its config at /tmp/mdadm.conf. I don’t know why, but probably there is a good reason

As others have said, /etc is writable during the installer.

More to the point: whatever other reason is likely bad. An app not using it's own config file in the standard location, losing data clobbering it's own config is, from experience (Red Hat used to do this a lot and our customers HATED it), a mistaken decision.

The installer filesystem should be an initramfs, so it should all be in RAM and writable (at least in the netboot case, but I don't see why CDs would do it differently; I expect the extra storage is just used for packages), at least for d-i stuff.

Even for proper (not install only) live CDs where the rootfs is really read only, I don't think I've ever seen a distro not mount an overlayfs on top so that changes can be made in RAM. Too much stuff breaks on your average distro if you really try to have a read only rootfs.

By choosing "expert" mode you can choose between any of the partition types (much more besides mbr and gpt). In standard mode it will pick mbr if none of your devices are bigger than the mbr partition limit (2TB?) otherwise it will pick gpt, without offering you a choice.

You can choose expert mode at the start when you boot the installer or you can switch to it from the menu once it's started.

I think you may be right about the other things, but this is not uncommon amongst OS installers. It is very hard to offer the full range of configurations available from all of the tools used to install a Linux system, without just putting the user at a shell prompt and letting them get on with it. Luckily that option is still there!

To remove the BBL from the devices that make up the array that the root filesystem is on you would need to boot into a rescue environment and assemble it there.

While the debian-installer does provide the rescue environment that you need to do this, it is by no means trivial.

Speaking as the article author I find it way way simpler to create the arrays from the shell at the start, rather than have to remember to reboot into d-i again, choose the rescue mode, tell it not to automatically assemble, drop to shell and assemble there. It's also faster — most of my servers take more than 60 seconds before even the serial-over-lan starts to display output!

That won't work in automated provisioning of "bare metal". It needs to happen automatically during mass-provisioning of systems.

To add to this, even though bad blocks are allowed by the contract between a device and the OS, because they are so rare, all code paths that deal with them are riddled with bugs.

For example, if you power off a disk improperly, all un-acked writes are allowed to contain a mix of the new data, the old data, or an unreadable sector.

If, on Linux, you now try to read one of those unreadable sectors, you will get an IO error - as designed. If you now try to write that sector, you won't be able to. Why? Well the kernel cache is 4k sectors, whereas the disk is 512 byte sectors, so to write a disk sector it needs to read the other nearby sectors to populate the cache.

That pretty much means every time you power off a disk improperly, you'll get a bunch of unreadable data and filesystem errors, even though the filesystem is journaling so should be able to cope with a sudden poweroff just fine.

Every hard disk has bad blocks and reserve space of good blocks to allocate from when one of these bad blocks is written upon. Also the drive's firmware has an internal list of known bad blocks discovered while a low level format was performed at the factory, which is why consumers should not perform a low level format ever again, unless they are aware of this and know how to write the updated bad blocks list back into the drive.

Either way, bad blocks on a hard disk are an unavoidable reality. The same is true for SSD's. It is normal and should be expected to run into bad blocks on the media during normal use of the hard drive, whereby the correctly programmed firmware will remap the pointer to a good block from the reserve and either tell the OS driver to attempt the write again (if the drive's write cache has been disabled, as it should be), or silently re-map to the good block and write it with the data from the drive's write cache. If power is lost during this, no amount of redundancy will save one. The only solution is ZFS or Oracle ASM.

It has always been this way, and will always be like this as long as we don't have media which cannot have bad "blocks", for lack of a better term.

As the article tries to state, the problem isn't that bad blocks can happen. The problems with md's bad blocks log are:

- Most of the time the entries don't correspond to actual bad blocks.

- It's buggy because it copies BBL between devices leading to another instance of "says the blocks are bad but they actually aren't"

- Once you've worked out that the entries are bogus it is still very hard to remove the entries or the BBL as a whole

- It's overly quiet in what it does. I monitor my syslogs but many people don't. There are many documented instances of people carrying entries in a BBL for years without knowing.

- Once MD thinks there is a bad block it renders that block on that device useless for recovery purposes. So in a two device array you just lost redundancy.

So this article has very little to do with the concept of bad blocks; it's about how md's BBL feature can be dangerous and why I don't want it enabled until it's fixed.