Garage S3: LMDB Corruption Post-Mortem

me.bertof.net

3 points·bertof··3 comments

> A power outage interrupted Garage mid-write on the LMDB metadata database. LMDB uses mmap’d I/O and, by default, operates with MDB_NOMETASYNC and MDB_NOSYNC (Garage sets metadata_fsync = false by default)

No, Garage disables LMDB's safety mechanisms by default. By default, LMDB performs whatever necessary syncs and is 100% crash proof, even in the face of power outages. Garage's problems are of their own making, not because LMDB has unsafe defaults.

TL;DR: LMDB is known to be vulnerable to corruption after an unclean shutdown. Garage provides a built-in snapshot mechanism (metadata_auto_snapshot_interval) exactly for this scenario. I hadn’t enabled it. I also had no Btrfs snapshots on the metadata partition. If you run Garage on LMDB, enable snapshots today.

I had a bunch of data disappear from a SyncThing share recently because SyncThing doesn't enable keeping backups of deleted files by default, and some bug caused it to think one client had deleted all the files and promptly propagated that "change".

I'm quite annoyed by such non-safe defaults. Make the defaults as safe as reasonable. If people want performance they can dig through the settings and turn off such settings.

I had something along these lines happen in the past. That's why I started using snapper on my home directory and the most important services. It saved my day already many times.

Reply on news.ycombinator.com