Hacker News Reader

Comment by mattbee | Hacker News Reader

The deduping in restic is just on the edge of acceptable for me, making me think I'd have trouble with a lot more data. Basically the one a month "prune" operation takes about 36h (to B2) . I feel I could be tuning something but also it works and I don't want to touch it.

View on HN

I backup around 2TB with Restic, also tried locally with Borg. The size is nearly the same. Sadly, I can’t even test with Tarsnap! (absurd pricing for 2TB).

> absurd pricing for 2TB

Well, it can't be that ba..

    $0.25 x 2000 = $500

Yikes. And this is without BW costs.

At $500/M you can just rent a dedicated physical server with a lot of HDDs and still have money left for your favourite pumpkin latte.

For comparison rsync.net says it's $0.015 per GB/Mo, for 2TBs that's $30/m and no BW costs.

Not in any way affiliated but I'm a happy user of Scaleway's Object Storage [0] together with S3QL [1]. It's not the fastest but they give you 75GB of storage for free so that's a fair trade [2].

[0] https://www.scaleway.com/en/object-storage

[1] https://github.com/s3ql/s3ql

[2] https://www.scaleway.com/en/pricing/?tags=storage

I'm renting $15/mo 2TB atom machine from OVH/kimsufi as second target for backups.

Now that I think about it... some kind of micro-distributed backup server (throw on few of your machines, auto-replicate between) would be a neat project...

It's not even that neat.

Just slap rsync/syncthing to the backup dir.

I do use syncthing on NAS + remote cheapo server for my day to day stuff, and bareos for rest.

It's just PITA to add another instance.

Curious how much you backup, which version of restic you're running and why you think the deduplication is borderline unacceptable. There were several major (orders of magnitudes) improvements made to pruning within the past ~1 year, that's why I'm interested.

A straight upgrade, that I can do :) It's been running for years without one.

I was only edgy about it because when it takes 36h it blocks the next daily backup, and I wondered whether that was going to get worse (it hasn't).

The max-unused percentage feature is well worth it to 80/20 the prune process and only prune the data which is easiest to prune away (i.e. not try to remove small files big packs but focus on packs which have lots of garbage).

In general, there's an unavoidable trade-off between creating many small packs (harder on metadata throughout the system, inside restic and on the backing store but more efficient to prune) versus creating big packs which are more easy on the metadata but might create big repack cost.

I guess a bit more intelligent repacking could avoid some of that cost by packing stuff together that might be more likely to get pruned together.

Reply on news.ycombinator.com