Swizzle Clogging

Yeah, I know. I thought swizzle-clogging as the "best way to trigger errors" was worth highlighting, an interesting technique.

> You'd probably have to roll your own tests that involve pulling the physical power plug on your servers, though.

I have some interesting failure mode/war stories from when I was setting up my home backup system.

I've got some external USB hard drives plugged into my media server. I figured it'd me smart to have a pair of archive disks that spend most of their life powered down and unwritable, to give me some chance of recovering from a cryptolocker type malware attack. So I have two big drives plugged into a pair wifi controllable power points, and some cron jobs that wake each them up late Sunday night (well, very early Monday morning) on alternate weeks, then rsyncs my Time Machine backups (stored on an always-on RAID1 pair) to the archive drive-of-the-week.

Of course the early POC cron jobs didn't do a sufficiently good job of checking error/return codes and rsync task completion, so occasionally my setup would indeed "involve pulling the physical power plug" on my hard drives... (I wasn't _too_ careful, since this would always happen on the alternating second backup of my Time Machine backup. While I got some file corruption in my archive drives, it couldn't have caused data loss unless both the original data _and_ the Time Machine backup also corrupted at the same time.)

FoundationDB is in use at Apple, Snowflake, multiple divisions within VMWare, eBay, Segment, Goldman Sachs, IBM, Epic Games, and more companies that are not public about their use.

Also, couchDB 4 will have foundationDB as storage engine.