Rclone: Rsync for cloud storage – CLI to sync between cloud storage providers | Hacker News Reader

Rclone: Rsync for cloud storage – CLI to sync between cloud storage providers

github.com

246 pointsgilad83 comments

Past related threads:

Rclone syncs your files to cloud storage - https://news.ycombinator.com/item?id=29213024 - Nov 2021 (8 comments)

Rclone – Sync files and directories to many cloud storage providers - https://news.ycombinator.com/item?id=22791036 - April 2020 (100 comments)

Rclone – Rsync for Cloud Storage - https://news.ycombinator.com/item?id=21906779 - Dec 2019 (3 comments)

Rclone: rsync for cloud storage - https://news.ycombinator.com/item?id=12398303 - Aug 2016 (129 comments)

Rclone: rsync for cloud storage - https://news.ycombinator.com/item?id=12049507 - July 2016 (1 comment)

Rclone – rsync for cloud storage - https://news.ycombinator.com/item?id=12045729 - July 2016 (3 comments)

Rclone – rsync for cloud storage - https://news.ycombinator.com/item?id=11404178 - April 2016 (1 comment)

Rclone is popular for good reasons. I have even replaced Dropbox with rclone nowadays. Why replacing Dropbox sync, one may ask?

Because:

* Rclone offers client side encryption (with a good algorithm and from a good library). Cloud providers usually offer server side encryption, but don’t want to offer client side encryption.

* You can mount the remote. Files don’t have to exist on client’s side to occupy space. This tool mounts anything!

* You learn one tool and use it everywhere

* It’s is under user’s control. You run it with a schedule that you want. I don’t have to run a closed source app continuously scanning my computer

* Robust, feature full, open source (especially for verification of encryption), large community for getting help, and under heavy development Rclone combined with Restic is how I manage my data in the cloud.

I use it for OneDrive on Linux. It's not as fast as Dropbox (which I used previously to OneDrive) but works great for backups.

> Rclone offers client side encryption

Can you view files in Dropbox website if they are encrypted locally by Rclone?

> You can mount the remote.

Do you have to mount the remote every time after rebooting your machine?

If I use Rclone to sync all dropbox files to my local machine, are they still accessible if my machine is offline?

» * It’s is under user’s control. You run it with a schedule that you want. I don’t have to run a closed source app continuously scanning my computer

A few years back, around 2016 I wanted to see how One drive syncing worked. So I set my IRC client (hexchat) to save logs on my OneDrive folder. If I remember correctly, I had hundreds of gigabytes uploaded to onedrive under a month. I think I had the default 15 GB or 25 GB storage iirc. My guess is every time someone said something in a chat, the file changed and one drive thought it was prudent to sync it with its servers immediately?

When a file changes, should we upload the entire file? Is it possible to somehow only transmit the diff and only send what has changed? Would it even make sense as most "normal" users are probably on mobile and diff has a cost in CPU and therefore battery, right? Or is CPU cheaper than WiFi for battery?

One of the features I like about Amazon.com Photos on Android is I can say only upload photos when charging.

> Is it possible to somehow only transmit the diff and only send what has changed?

This is what rsync does.

edit this refers to rsync the software, not rsync the company that provides storage.

Really? How does it know what the diff is when it only has the new version of the file?

https://en.m.wikipedia.org/wiki/Rsync#Algorithm

Also popular for some unfortunate reasons as well, it is becoming the exfil and sync tool of choice for ransomware operators.

context : first hand experience in IR

Slight tangent: I always found it to be an added plus/bonus to a tool if it can survive being scrutinized by well-formed and long-lasting groups.

What would be my preferred VPN/Mail/Payment/Storage/Encryption tools?

The one that has been scrutinized by TLA to catch these groups. On a long enough time scale, it can expose if they truly are "no logs", "fully encrypted", "privacy-conscious" or if it's just marketing buzzwords when they actually face a warrant.

Rclone is fantastic. It's much easier to use that most of the cloud storage APIs themselves.

Plus rclone makes it easy to migrate between providers. I just moved a backup process from Gcloud to B2 now my side business is growing up: you can start with whatever storage is free/convenient then switch to something more mature later with minimal effort.

Hear hear.

Been personally using for 1.5 years with Backblaze thanks to HN with 0 problems. Have even done some restores succesfully!

I run a ~75TB plex server with 99% on Google drive with 0 issues using rclone mount on a haswell i3. Can get close to 20 folks on gigabit before it blinks.

Are you using the VFS cache liberally? I'm quite stingy with my settings, but get downloadQuotaExceeded errors from time to time.

No rclone cache. 2 week local storage cache with unionfs

Thats an impressive setup. Is that across multiple g drives?

Probably a single user business that's has unlimited storage. I think the plan is for $12?

The only Google Workspace plan that offers unlimited is the contact-sales-for-pricing "Enterprise" tier. The $12/user/month plan only includes 2TB/user: https://workspace.google.com/pricing.html

$20 a month. Enterprise with 1 user.

> $20 a month. Enterprise with 1 user.

Holy shit. I didn't know that was an option. I just assumed that there was some minimum number of users (like 10-20) that made it non-viable for personal use-cases.

there is and there isn't. it says minimum 5 users (so $100 a month) but they don't enforce it.

Here is a keystroke-by-keystroke HOWTO that we wrote last year:

https://rsync.net/resources/howto/rclone.html

and although the example is S3 <-> rsync.net, it is not necessarily rsync.net-specific.

You could use this HOWTO/workflow for any two targets.

Do you contribute code to rclone?

FYI You can choose the type of storage by using the short name instead of the index

    4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
      \ "s3"

"4" is fragile because future changes could change that offset (it happened at least once with Google Drive), but "s3" is fixed.

Thanks! I will pass that along and see that it is "fixed" ...

Thanks for this write-up, it's what finally got me over the activation energy barrier to setting up backups of a small home-automation setup I run.

The rclone docs are very extensive but having a short, simple example was the kick I needed to get past the myriad options.

I like rclone for the wide array of storage options [1] plus a special crypt remote [2] that provides client-side encryption for any of the other storage options. There is also a browser GUI you can spin up locally if that floats your boat [3].

[1]: https://rclone.org/overview/

[2]: https://rclone.org/crypt/

[3]: https://rclone.org/gui/

Nick Craig-Wood (ncw) was the CEO of the last company I worked for, and I consider it a privilege to have worked for him. He's one of the smartest people I've ever met, and an incredibly nice person too. I have many fond memories of the annual staff barbecue in his garden.

He reviewed my first Go PR a few years back when I posted something new for rclone. He's super nice, indeed. Kudos Nick!

I worked with nick on cloud storage systems, he is one of the nicest guy ever

I attended his talk on deadlocks in Go at Gophercon UK this year. It was great, he showed some real life examples from rclone. Here is a link if anyone is interested - https://www.youtube.com/watch?v=9j0oQkqzhAE

I once made a software distribution system for software with thousands of files in a huge directory tree using rclone. I'd upload it all to S3 and then clients could run an update script that used rclone to sync just the changed files. No server needed. A better solution probably exists but I couldn't find any other system at the time that could work directly from S3 or any other dumb file host and still selectively sync only changed files.

I recently noticed in the rclone source tree[0] that they are working on making “librclone”. I would love it if they exposed the rclone internals as a library as the only other solutions (e.g. bita, zsync, librsync) still require a fair bit of tedium to use as a generic application updater.

[0] https://github.com/rclone/rclone/tree/master/librclone

Since we're on the topic: Which cloud provider do you recommend for backup of personal files?

Depends on the amount you have.

There is only one provider that has rclone built into the platform:

  ssh user@rsync.net rclone sync s3remote:bucket path/path/dir

So you can transfer directly between endpoints without using your own bandwidth.

In fact, you don't even need to install rclone yourself - just run it, over SSH, at rsync.net.

Here is a broad overview of using rclone with your rsync.net account:

https://www.rsync.net/products/rclone.html

$100/TB/month is significantly more expensive than the standard object storage options. Backblaze is $5/TB/month, for example.

For personal file backup, you only need a fairly reliable object storage system. And Backblaze seems to be the cheapest.

Standard pricing is $25/TB/month.

We have had a "HN reader discount" for many years ... just email and ask.

If you're optimizing for price, I would suggest Backblaze.

1fichier.com trends to €1/TB/month.

The website is not reachable. Maybe that’s why it costs that much? Or it’s not .com?

The real site is https://1fichier.com , works for me and always has, check any blocking things you have? People use them for somewhat sketchy things, but they’re a legit if somewhat eccentric French company that’s been around for years (and is specifically supported by rclone: https://rclone.org/fichier/). Sure, don’t put your critical data there without a backup, but at the price, it’s a nice option to have.

There’s a .info that shows up in search results that I think is an otherwise unaffiliated reseller that they haven’t bothered to go after.

You are right. It worked on vpn. So blocked by local authorities then (it’s piss easy here for anyone with a law degree or half).

By the way they seem to offer quite some storage for free! :-o How do they sustain? and even paid prices are unreal.

Also I failed to understand whether they’re a file sharing service or file backup?

There's "optimizing for price", and then there's you being five times more expensive than Backblaze.

Hetzner storage, which is comparable to your offerings (supporting a dozen or so methods), is 9.4 euros per month per TB.

How do you justify being so much more expensive?

Well for one they offer options on more than 1 continent.

They don't charge for traffic, it matters if you have >10TB, but not individuals taking backups.

Wow, I'm not sure how I got the pricing so wrong. I remember visiting the rsync.net pricing page to check. I'd edit/delete my GP comment if I could.

To be clear, is the $25/TB/month quoted with or without the HN discount?

$25/TB/month is the headline price for "everybody".

The HN discount price is lower than that (by a fair amount).

The downsides to the discounted plans are as follows:

- you can't have subaccounts/sublogins (child accounts).

- we won't give you technical support for borg/attic/rclone integrations. We assume you are an expert. Of course you get normal run of the mill support ...

If you consider storage in isolation, Backblaze does seem to be the best deal. However, there may be bundles that include storage that are a better deal if you need the other things in the bundle.

For example if you have sufficient need for Microsoft Office programs to justify a Microsoft 365 subscription, that comes with 1 TB of OneDrive on the Personal subscription ($6.99/month, $69.99/year).

Most backup programs seem to know how to work with OneDrive directly or know how to use rclone which knows how to work with OneDrive [1].

If you need more than 1 TB, OneDrive becomes less attractive. It seems you can add another 1 TB for $9.99/month, but unless you have to all your storage be in one place, it would be cheaper to get your second TB at Backblaze. (And if you need 2 TB in one one place, buying 2 TB at Backblaze + An MS 365 Personal subscription would be the same price as an MS 365 Personal + 1 TB Extra OneDrive, and the former gives you your 2 TB in one place at BB and 1 TB at OneDrive).

The Family subscription ($9.99/month or $99.99/year) includes 6 TB of OneDrive, but it is partitioned into 6 separate 1 TB shares for 6 people on the account. A person is identified by an email address, so there is nothing stopping you from making 5 fake people, and one person can give another write access to folders on their OneDrive, so you could cobble together a scheme to use all 6 TB but there would be some hassle.

I tried doing that once, having the fake person give the real me write access to a folder. I then uploaded a bunch of files to that shared folder from my regular account. Shortly afterward the fake person got disabled for TOS violations (but could be reenabled via a simple automated process). I'm not sure what the issue was, but what I uploaded was a bunch of Springer ebooks to from the large collection that Springer made freely downloadable early during COVID so that I could read them on my iPad. It could be that triggered some sort of anti-piracy measures.

If you can find a way to make it work, though, 6 TB for $9.99/month or $99.year is a good deal even if you don't need the Office apps.

[1] https://rclone.org/onedrive/

Right now I'm using Dropbox, One Drive and iDrive, none of which are that great. I've looked at tons of others.

Your prices aren't offensive, but they're not cheap.

I have a need to store several terabytes of PDFs - mainly old magazines that are in the public domain, or where the copyright is essentially lost. Are your systems going to scan every item that gets uploaded and then erase my account and all my files if one PDF comes up as having a current copyright issue?

I ask in all seriousness because this happens regularly to those of us on

https://www.reddit.com/r/DataHoarder/

see, e.g.

https://www.reddit.com/r/Degoo/comments/iixlse/degoo_copyrig...

"Are your systems going to scan every item that gets uploaded and then erase my account and all my files if one PDF comes up as having a current copyright issue?"

This is who we are:

https://www.rsync.net/resources/notices/canary.txt

We were the first provider to publish a Warrant Canary and it appears we're one of the few that still update it.

So, no.

No, we do not scan accounts for any purpose whatsoever.

Please review our extremely short and simple privacy policy:

https://www.rsync.net/resources/regulatory/privacy.html

... and do be in touch about the HN readers discount.

EDIT: rsync.net has, in the past, donated a lot of resources to Jason Scott and Team Archive for archival purposes just like you are describing. I am also, personally, a sustaining donor of Internet Archive. All of this to say: I am very interested in this archival work you are doing and would like to help support it - please do be in touch ...

I've been curious - has the warrant canary idea been endorsed by anyone's legal counsel? I can't tell if it's a real legal loophole, or a letter-of-the-law thing that a judge would treat as equivalent to revealing a warrant.

Seems like it's an open question.

> In September 2014, U.S. security researcher Moxie Marlinspike wrote that "every lawyer I've spoken to has indicated that having a 'canary' you remove or choose not to update would likely have the same legal consequences as simply posting something that explicitly says you've received something."

> ..That said, case law specific to the United States would render the covert continuance of warrant canaries subject to constitutionality challenges.[citation needed] West Virginia State Board of Education v. Barnette and Wooley v. Maynard rule the Free Speech Clause prohibits compelling someone to speak against one's wishes; this can easily be extended to prevent someone from being compelled to lie.

https://en.wikipedia.org/wiki/Warrant_canary

Is it possible to use rclone as a backup tool? With the usual features like compression, e2e e, de-duplication, incremental.

I don't think so, but HashBackup (I'm the author) has all that and has an interface to rclone if you use a storage service that isn't supported natively.

Looking for personal backup and with GUI. Here https://news.ycombinator.com/item?id=29446135

We've been telling people to use 'borg' for several years now and it appears to have become the de facto standard for just what you're describing.

It has been described as "The Holy Grail" of backups:

https://www.stavros.io/posts/holy-grail-backups/

I use borg indeed, via Vorta. Also, I am a rsync.net customer (I think you should start with a smaller min storage for pay what you use, or keep it pure pwyu).

I am looking for another backup tool that backs up to an S3 storage destination and has a decent GUI (restic doesn’t have a GUI; relica doesn’t count; and borg doesn’t support s3). Since rclone has an experimental GUI I wondered.

Thank you for the excellent reply. Just the fact you are on HN answering queries is a reason alone to buy your services.

> https://www.rsync.net/resources/regulatory/privacy.html

It says there that you don't use cookies, but to clarify, I assume you set a cookie when a user logs in?

I like Backblaze because it works well on my Mac and I don't have to think about it. That's about all I want!

Backblaze B2 has free ingress and egress (w/ Cloudflare) and the cheapest pricing that I know of per GB, but it looks like CloudFlare's block storage might be a competitor soon.

B2 as a backup solution it's a compelling option as far as pricing goes.

I'd offer a warning that I have used B2 for highly available blob storage before (2-3 years ago) ... it didn't work well as traffic scaled. I'm not sure if that's the case now.

For anyone wondering about B2 encryption. I looked at B2 last year and ultimately didn't use it as they didn't have encryption at rest [1]. It seems they've now added that [2].

[1]: Client-side encryption would have needed quite a bit of engineering to make work for my use case.

[2]: https://www.backblaze.com/b2/docs/server_side_encryption.htm...

Duplicati and a few other self-hosted backup tools provide great client-side encryption and use rclone under the hood for syncing to B2.

I'd like to piggyback here and add a second requirement: I'm currently looking for a storage provider that supports pulling backups from the outside. Kinda surprised this is not more of a topic, as it should be a simple way to make backups more robust against compromise?

Seems rsync.net supports this (see sibling comment). Any others?

Whatever provider you choose, make sure you encrypt your files locally to avoid the scenario where the provider blocks you because the AI decided one of your files is a virus, a copyright violation or matches some image hash.

There is no need for your backup provider to be able to look into your files so there’s only downsides to allowing them to.

I've been using Backblaze B2 since it launched, and HashBackup (I'm the author) was one of the first B2 integrations.

What I like most about B2 is the transparent, no gimmick pricing. No monthly minimum, no minimum file size, no paying for storage for 90 days after you delete a file. And they give generous free allowances: 10GB/mo (forever, with no credit card required), 1GB of free egress per day, and up to 5000 free transactions per day.

I backup the HashBackup dev server to B2 every night, around 80GB with mostly VM images, and the total backup size while keeping the last 30 days + 12 monthly backups is around 50 GB and costs 25 cents/month. They bill me every few months, I guess when enough charges accumulate.

Here are the backup stats for the HashBackup dev server:

               3,617 backups
              334 TB file bytes checked since initial backup
               80 TB file bytes saved since initial backup
                 910 total backup hours
               79 GB average file bytes checked per backup in last 5 backups
               23 GB average file bytes saved per backup in last 5 backups
              29.78% average changed data percentage per backup in last 5 backups
             13m 43s average backup time for last 5 backups
                 637 archives
               55 GB archive space
              84.11% archive space utilization 46 GB
               810:1 industry standard dedup ratio
              9.5 MB average archive space per backup for last 5 backups
              2465:1 reduction ratio of backed up files for last 5 backups
              234 MB dedup table current size
           9,442,920 dedup table entries
                 48% dedup table utilization at current size
             724,059 files
             671,701 paths
          10,550,610 blocks
         127,965,672 block references
         117,415,062 deduped blocks
              12.1:1 block dedup ratio
              5.2 KB average stored block size
              616 GB backup space saved by dedup
              49,648 average variable-block length

I'm not sure why this bothers me so much but it does not "sync between cloud storage providers".

I don't really know how this misunderstanding is so pervasive, even among the HN crowd, but no, no amount of magical code can make it possible to directly migrate between S3 and Google Drive. It's going to download, and upload to sync.

This is even reflected in the project's description line, which is more accurate than the current HN headline: Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers.

That having been said, it's of course invaluable software. Mounting 10TB of Google Drive into a Raspberry Pi running Plex is still something that feels like it should be harder to accomplish than it is.

Does something the word "sync" require that no intermediary hardware must exist between the endpoints? And does this definition disqualify the use of devices like, say, routers and switches?

>no amount of magical code can make it possible to directly migrate between S3 and Google Drive

writes magical code that fires up an unmetered-bandwidth billed-hourly VPS for a few hours (Scaleway is nice)

This was exactly the comment I was looking for. I've had to move a lot of data to GDrive from Dropbox before and have been keeping an eye out for tools to help with this.

Using Google Colab is the best way I've found so far - mount your GDrive and you can wget to it directly from Dropbox (or anything else).

I'm using rclone beta right now for its bisync feature. This was the last missing piece that I badly needed to synchronize my notes. And I'm very happy that it's finally implemented. Kudos to all rclone contributors!

That sounds cool! Is there some preview documentation available somewhere?

Rclone is excellent, I use it to deploy my static site to Neocities and it's worked excellently for ages.

Love this tool. Along with Golang's Excelize library I use it to build a bunch of tools

Love this software, use it for everything from mounting cloud storage to syncing backups

Rclone is great! All of my duplicati backups go to S3, then in the run-script-after script rclone copies them from S3 to B2.

I have used Rclone for both personal file backup and production data. It is dead simple and amazingly robust.

I wonder if I could use it with a rsync endpoint? It's not listed in the backends.

Past discussions:

https://news.ycombinator.com/item?id=12398303 (5 years ago)

https://news.ycombinator.com/item?id=22791036 (2 years ago)

Is this a good multi threaded local/ssh alternative to rsync?

It is. Another good alternative is lftp.

“Rclone” and not “Csync”? Missed opportunity! ;)

Does it support S3 Glacier?

Yes, you would have to been pretty obscure to not be supported. https://rclone.org/s3/

Reply on news.ycombinator.com