Syncer: fast stateful file/disk data syncer written on Go | Hacker News Reader

Syncer: fast stateful file/disk data syncer written on Go

github.com

56 pointsstargrave30 comments

I assume the title has a typo and that it's supposed to say "written in Go".

With that said, as a user of a tool/library/service, why should I care what language it's written in?

I want to know if a data-syncing tool is reliable or not. I know rsync is, but OK that's not always super-duper fast.

I know volume-streaming with btrfs and ZFS works wonders, without the need to do manual book-keeping of a secondary state and hoping you ALWAYS keep that secondary state up to date. I doubt this out-performs the FS just doing the job itself, and it sounds much more prone to user error.

So what does this offer over those? It sounds like it has less features and is not as reliable. Being written in Go offers me nothing I don't already have.

I answered in another comment on that question. The main point of course is not a language. ZFS incremental snapshots and its send/receive are great, but my task is to create binary identical image of hard drive with bootloaders, partitions and so on. I have got SSD that can breake and slow USB connected HDD. I want to sync them several times a day and be able to trivially switch them in the case of failure. ZFS is only a filesystem. I need to prepare harddrive with bootloaders to use ZFS send/receive. I do not want it. Raw dd is what is needed, but it is slow -- the only reason this utility is written.

It should be enough to copy partition table and boot loader only once. After that, it's safer and probably faster to backup with consistent btrfs/zfs snapshots. Copying raw blocks from a writable mounted filesystem will lead to inconsistent backups.

Agreed that it is preferred way, after raw dd with unmounted filesystem. But I need to be prepared for that: I am a human, I can forget some steps. Here command takes only two arguments (src and dst) -- simplicity is reliability against humand mistakes. Some data will be corrupted, but with journaling filesystems or something like ZFS this is not dangerous (unfinished transactions will be ignored).

> ZFS incremental snapshots and its send/receive are great, but my task is to create binary identical image of hard drive with bootloaders, partitions and so on.

If you use UEFI, bootloader is just a normal file on a normal (FAT) partition so no special magic is needed, but fair enough.

ZFS typically handles partitions its own way so no need to sync that.

But really: If you just want to keep to full drives in sync, at block-level so one can at any point be a stand-in replacement for the other, it sounds like what you want is RAID0. Isn't that what you want?

And with UEFI you can boot those RAID-volumes, even though they are soft-raid volumes.

Not saying your need doesn't exist, but that existing solutions which are more standardized and requires less administration seemingly already exists.

I believe if you raid0 an HDD and an SSD you lose fast writes because the OS waits for it to be written to all disks in the array.

Maybe depends on the implementation, I I'd imagine it would degrade performance.

Forgive me, it's Monday morning and I haven't yet had my coffee, but it sounds like you're describing a block-level copy between two drives, something which RAID0 is most definitely not.

Unless I'm misinterpreting, I think you were going for RAID1.

In my experience, btrfs send does a lot of random reads and is therefore rather slow when reading from mechanical disks. Backing up to a mechanical USB drive is fast enough, restoring from that drive can take forever (but you can use seeding). This tool reads the disk sequentially and should reach much higher throughput.

On the other hand, btrfs send skips empty and unchanged blocks. If the disk is mostly empty, send could still be the faster option.

But btrfs is supported on GNU/Linux platforms, ZFS on FreeBSD/Solaris ones. So they are not portable, OS-specific -- and it can be disadvantage.

Zfs-on-Linux has come a long way. I've been running it for my storage array at home without issues for about a year.

well I take 'written in go' is more than a warning than a bonus at this point, especially after looking at some similar tech and finding all this https://github.com/syncthing/syncthing/issues/9 (ended up using glusterfs in the end)

besides, this lib only cover one sync scenario and only one and does it only on demand. might be fast because it stores the hash somewhere, but that also means you'll run in issues if the mirrored slow drive content changes by any mean.

I don't see what your issue is with syncthing. syncthing is meant to be cross platform, so using inotify was an obvious problem.

Plus syncthing is an entirely different use case from glusterfs. Think Dropbox, not one-filesystem-across-many-machines.

None at all with syncthing. It's awesome, but doesn't cover my use case. And symcthing does say "Syncthing replaces proprietary sync and cloud services with something open, trustworthy and decentralize" - except as you say it doesn't.

You can get inotify-or-equivalent support in Go for specific OSes reasonable well with bindings (simple enough even to write your own if push came to shove), what's missing is a cross-platform, simple API. Which, arguably, is because it turns out that despite the problem sounding simple, a cross-platform, simple API is all but impossible. You can sort of get one for a single file or a single small directory with no recursive subdirectories, but beyond that, you're surprisingly boned.

This hasn't got much, if anything, to do with Go specifically; in particular, I've been aware of everything I write in this post since before I was even aware of Go. The OS support is just too variable and shot through with subtle-yet-critically-important semantic differences, in addition to the fact the problem is harder than it looks at first. Trying to write a lowest-common-denominator API leaves you with a really low LCD. Trying to write a higher level API that offers the more convenient stuff leaves you with an API that makes it trivially easy to write something that works really, really well on one platform and is completely non-performant on another (in particular, trying to automatically "recurse" on the directories). It turns out there's no happy middle here for the general case.

>With that said, as a user of a tool/library/service, why should I care what language it's written in?

Because it tells you a lot with what other ecosystems is compatible, how much of a fuss it would be to interface with it (in case it's a library and you use e.g. Java, you'd normally prefer a pure-Java implementation instead of a C+JNI one, etc), if it's a good fit for your existing dependency management, if you know the language to go in and fix an issue or adapt it, etc.

It would therefore also be useful to state the platforms it would work on - in this case it seems to require a block-level device and is unlikely to be usable on Windows.

> With that said, as a user of a tool/library/service, why should I care what language it's written in?

It makes sense to mention it for people who for example for learning purposes want to read code written in Go.

HN is not really a "new products for you to use" site, and I find it a bit strange when people read and reply to posts purely from the perspective of a user or potential user.

If you approach the post with a more open curiosity I think it will make more sense to you why the language is a relevant detail, even if it still doesn't interest you personally.

It's late and maybe I missed something looking through the code, but what happens if you don't have write permissions on the directory that the statefile is in? The copy operation is performed but then the state is lost when the statefile rename fails.

At first, temporary file created in current directory. So if directory is not writable -- temporary file creation at the very beginning will fail.

Ah, that's what I missed. I guess there's still a possibility of permissions changing during the transfer, but I don't think there's anything you'd want to do to handle that other than error out.

In that case I will print the message telling that something went wrong but temporary statefile is alive: https://github.com/stargrave/syncer/blob/master/syncer.go#L2...

Suggestion: Don't set a default for dst. That looks really dangerous.

Agreed. Fixed. Thanks for the suggestion.

A former colleague has written something quite similar[0] few years ago, which we used to backup encrypted disks (actually snapshots of encrypted disks to be precise).

[0] http://chunksync.florz.de/

Nice, but in which use cases it may be useful?

I have got fast SSD that can break down every day. I have got USB hard drive. I want to sync data every day, twice a day between them as fast as possible (raw dd takes an hour). If SSD is broken then I just plugin USB hard drive and continue working (with some minor data lost).

ZFS has incremental snapshots, it is fast, but I need to restore bootloader, partitions and everything corresponding to be able to bootup and work again.

Scratching your own itch is the correct path to success, in my books. Keep up the good work.

this is great. I wonder how it would perform when I need to sync 10mil. files. Rsync and anything else breaks down at this point, well it does work but it takes forever and rsync bloats to 10-20GB memory foot print and after a few hours starts streaming updates.

syncer works on block level, with raw byte sequences. It knows nothing about file systems. Everything is limited with sequential read/write speeds of your hard drives.

It takes blocksize-size memory block for each CPU in your system. I have got 4 CPUs and work with 2 MiB blocks: program will take 8 MiB of RAM.

Reply on news.ycombinator.com