Hacker News Reader

Comment by rwmj | Hacker News Reader

It's a shame that Linux doesn't have APIs as comprehensive as Windows VSS. One of the several features is something called Volume Shadow Copy Service which lets you take backups of block devices which are in use. It's kind of similar to this but more supported and I think it also interacts with user processes like databases, quiescing them so you can take a consistent snapshot.

[Also if you are playing with custom block devices in Linux -- mounted or otherwise but not /dev/sda -- https://gitlab.com/nbdkit/nbdkit https://libguestfs.org/nbdkit.1.html One filter we have which is kind of similar to blktrace is called the nbdkit-stats-filter https://libguestfs.org/nbdkit-stats-filter.1.html]

View on HN

The dattobd [1] block device driver fills this gap. It basically implements point in time like behavior on Linux, with hooks similar to the ones that VSS provides. It is used in the Datto Linux backup agent in combination with the backup appliance [2], but it can be used stand alone as well (see readme).

Disclaimer: I'm a software engineer at Datto.

[1] https://github.com/datto/dattobd

[2] https://www.datto.com/products/siris

This is just the coolest thing in the world, thank you so much for open sourcing it.

Have you asked upstream what they think of it, at all?

Hey, I'm one of the folks that work on maintaining dattobd. We haven't had a chance to talk to the upstream Linux kernel storage folks about a way to upstream it, but we're interested in exploring it, of course!

Once the world gets more back to normal, you could submit a talk topic to LSF/MM. Or just email the appropriate mailing list.

I've been doing my full disk backups with zfs snapshots for a while now and zfs sending them offsite. Granted it's a freebsd box, but same tools should be available on Linux.

Definitely possible on Linux. In the not distant past I had a large on-prem PostgreSQL DB that had a hot standby that we used for backups. Since ZFS snapshots are atomic you can just snapshot the dataset under the DB without worrying about quiescing the database. And then zfs send backups to files that are then backed up offsite.

If you ever cared it's easy enough to clone one of those snapshots and bring it up as another DB instance to poke around at how things used to be...

I've been doing this with several production PostgreSQL instances.

PostgreSQL on ZFS is great.

I have zstd compression enabled and I average 3.50x compression ratios.

(Probably some pretty awful CPU tradeoffs in there, but system load and query times seem fine. My database is 50 GB before zstd compression, so enabling it helps a ton.)

I also have ZFS snapshots taken hourly and shipped offsite. It's awesome that I don't need to pause PostgreSQL or anything to take the snapshot. It's atomic and instant.

With some quick and dirty `time` style tests zstd has a pretty low overhead. IIRC writing was around off = 100, lz4 = 110, zstd = 115 cpu utilization on my personal data set that resulted in 1x, 1.7x, 2.1x compression. Reading was negligible, single digit percentages, for both lz4 & zstd. For anything on a spindle that's a pretty good trade off of CPU time.

VSS is pretty great. I use Macrium Reflect on Windows, and it just works. I've never once had a problem restoring a Macrium backup to bare metal, even if it has a bunch of incremental backups.

VSS is great indeed. Very useful to eg. P2V a machine live, even saving the P2V image on the same drive.

But VSS, like LVM or other snapshoting FS, requires dedicated FS support. You don't have VSS on FAT32. Here, blktrace works at block device level, so does not rely on any FS support.

I'm not sure I follow here. Linux does have LVM, what's the problem with it?

That's also what the article says: If you have LVM, you don't need any special tool.

LVM doesn't signal to databases and other user applications so it's definitely not a replacement for VSS. As for whether LVM is a replacement for nbdkit, I guess it depends on if you like writing kernel code, or prefer instead to write nbdkit plugins in userspace in a variety of programming languages (even shell script: https://www.youtube.com/watch?v=9E5A608xJG0)

For user application it does not signal anything I guess, but I'm not sure if that's important for databases. Databases are usually crash-consistent, which is equivalent to a snapshot.

However, it does signal to the filesystem, quoting https://manned.org/fsfreeze.8 :

       fsfreeze is unnecessary for device-mapper devices. The device-mapper (and
       LVM) automatically freezes a filesystem on the device when a snapshot
       creation is requested. For more details see the dmsetup(8) man page.

No one is doubting that databases are crash consistent. But if you read the other replies in this thread you can see there are still advantages to doing the backup in cooperation with the userspace programs, eg in time taken to restore instead of having to do full WAL replay. That's ignoring the non-database cases too.

I think I saw Oracle mentioned earlier, this has some info about the VSS integration. It's called 'Oracle VSS Writer': https://docs.oracle.com/database/121/NTQRF/vss.htm#BEIJJCAE

I guess MS-SQL does something similar, and that does seem useful beyond what LVM does.

Also VSS apparently can offload the snapshotting to a SAN ('VSS Hardware Provider'): https://docs.microsoft.com/en-us/windows-server/storage/file...

That can't be supported by many databases, it must be insanely difficult to get that right, is it really used?

It just needs to listen to explicit fsync commands and choose a matching point-in-time. Beyond that, no, it's not hard as a database to tell the OS about moments where a block-level disk image would not need repairing, after the OS asks you.

I assume this is to prevent situations that would need time-intensive WAL-replay or such, and there it'd only be opportunistic with the "sudden power loss" recovery as a fall-back path.

Oracle, MSSQL and some rarer ones (Actian Zen?) are "VSS-Aware" in that they register a writer driver. This gives them a callback before the snapshot starts - a chance to flush transactions and hold off writing any more, for a second or two until the disk snapshot is created and block tracking has begun. Then they get a second callback saying they can resume.

More interesting is e.g. Hyper-V using this to snapshot the guest VMs.

It just takes you one step beyond being only crash-consistent, so that everything is actually application-consistent.

Implementing a VSS writer is a big 90's-style COM interface, but there is also a lightweight alternative model on top of that that eludes my search right now.

Red Hat has some similar convention for freeze/thaw scripts that can make API calls to the DB to ask it to do the same thing.

Does it just buffer to memory during this, or do the databases block writes?

> "More interesting is e.g. Hyper-V using this to snapshot the guest VMs."

It can run through VMware tools into virtual machines, too, and down a layer to SAN storage and trigger a SAN snapshot so that a backup system like Veeam can take data from the SAN while knowing the guests two levels higher flushed data before the snapshot was taken.

or e.g. here Nimble SAN can talk up to servers through VSS to trigger SQL Server quiesce before they take a SAN snapshot: https://infosight.hpe.com/InfoSight/media/cms/active/public/...

Postgres supports backing up by just copying its files if you let it know using pg_start_backup/pg_stop_backup.

https://www.postgresql.org/docs/8.1/backup-online.html

These database already have to make sure that they can recover from the system crashing. So it is probably only incrementally more work to add a feature to write out some extra metadata to enable these sorts of online backups.

AIUI, Postgres rotates its WAL, so once something is acknowledged as having hit the storage, the WAL entry referring to that write can be thrown away and that WAL space can be re-used. When you do pg_start_backup, it just temporarily stops throwing away old WAL entries, so that the copy of all the files you make elsewhere is guaranteed to have all the WAL entries it needs to make the database fully consistent.

Is VSS deprecated? They took out the "previous versions of file" feature in Windows 8.

Apparently the GUI part was removed in Windows 8 and then re-added in Windows 10: https://en.wikipedia.org/wiki/Shadow_Copy#Windows_8_and_Serv...

Reply on news.ycombinator.com