Move Docs: https://rclone.org/commands/rclone_move/
S3 Docs: https://rclone.org/s3/#amazon-s3
Supports parallel server-side copies and deletes (no server-side moves, unfortunately) so this would have been much faster.
So I could, for instance:
ssh user@rsync.net rclone s3:/some/bucket gdrive:/blah
ssh user@rsync.net rclone s3:/source/files /my/rsync.net/account
Is this too good to be true?!
Thank you for sharing! Yet another gem found in HN comments.
20 years ago, I was on irc.lightning.net and thought that was the best run efnet server. The MOTD advertised their IP transit services. They became he.net and that is why rsync.net does (most) of their IP transit with them.
The lack of bandwidth charges sounds really nice but [1,3] have that too and unlike [1,3], Rsync.net doesn't support protocols that are suited for bandwidth-intensive applications (like serving large files to end-users). It only supports SSH-based transfers [4].
[0]: https://www.rsync.net/pricing.html
[1]: https://wasabi.com/cloud-storage-pricing/
[2]: https://www.backblaze.com/b2/cloud-storage-pricing.html
[3]: https://www.hetzner.de/dedicated-rootserver/matrix-sx?countr...
If you look at the pricing on that rclone page[1] you'll see that at quantity it is as low as 0.5 Cents Per GB / Month.
As for bandwidth intensive protocols, we have HPN-SSH[2] patches built into our environment which allow for high bandwidth transfers, over SSH, over long WAN links.
That's good but at the volume needed to reach it, I imagine that you're in the realm of "contact us and get a special deal" at other providers (e.g. AWS, GCP, Azure) too.
Plus, we haven't even got to the "you have to pay for capacity, not usage" aspect of rsync.net pricing.
Pricing is the big thing I wish you'd fix. Your service has a bunch of cool features but they're just not worth that price, not when I can get most of the same features from Hetzner for a fraction of the cost.
> As for bandwidth intensive protocols, we have HPN-SSH[2] patches built into our environment which allow for high bandwidth transfers, over SSH, over long WAN links.
That's cool (honestly, not being condescending) but my point was that for most of the reasons you'd be excited about free bandwidth, rsync.net isn't a viable solution. For example you can't host images, videos, software updates, machine learning datasets or other large binaries and serve them over HTTP on the public internet.
Correct. We don't do these things and we never will.
When you 'nmap' an rsync.net storage array, you get:
22 TCP
... and that's it. There are no other services running, or offered. There are no interpreters in the environment. The filesystems are mounted noexec,nosuid.
We do one very simple thing and that's it.
But you don't.
To be competitive you either need to drop prices or do more stuff to be worth the money.
But having no other cost than the storage itself is relieving and having the service running for nearly 20 years gives you a peace of mind on their reliability. I also hear their support is good
I would want a bit more modern web interface though. BorgBase does look interesting to me, except only 2 years of operation can't tell about its reliability and longevity.
With sufficient --transfers and --checkers values you can easily saturate 10-20 gigabit/second links, so with a handful of machines you can transfer 25TB in about an hour, no engineers required.
Worth noting that rclone also supports many other storage providers or even plain SSH, so it pretty much obsoletes rsync. It also has many other nifty features useful outside of the "sync files" use case.
In this case S3 will be the bottleneck. 25 TB will have around 700M files (assuming it's as dense a linux installation). S3 can do 5000 operations per second. It'll take at least 40 hrs (assuming everything works at peak speed during the whole process).
Ramp up the parallelism, and don’t make 7 people monitor it round the clock.
I saw in the Reddit discussion that one can set up role/privileges (as expected), use the (new) user/role/group X for the move, and cut all others.
I understand that the original planning went bust 2h --> 48h, but that it buffled me was the "It sounds more like you had 2 hours for the delete part of the operation."
S3 replication and email to AWS supports to apply replication for existing object(per reddit discussion) seems a better method.
That trash behavior isn't specific to rclone and probably something on Google's end, but rclone is what I notice it with most.
The exact wording of the relevant section of the mail:
We are writing to let you know that starting October 13, 2020, Google Drive is making a change so that its trash behaves more consistently with the rest of our G Suite services with regards to automatic deletion. This means that any file that is put into Google Drive’s ‘My Drive’ trash will be automatically deleted after 30 days. Items in trash will still continue to consume quota.
Please note that starting October 13, 2020 any files already in a user’s trash will remain there for 30 days. After the 30-day-period files that have been in the trash for longer than 30 days will begin to be automatically deleted.
What does this mean for my organization?
Any file that has been in the trash for longer than 30 days after October 13, 2020 will be automatically deleted. We will be showing in-app messaging in Drive starting September 15, 2020 and in our Editors products (such as Google Docs and Google Forms) starting September 29, 2020.
A few things to note:
* Files in shared drives trash are already automatically deleted after 30 days. * These changes affect items that are trashed from any device and any platform. * Files deleted from Drive File Stream will be purged from the system trash after 30 days. There is no impact to Backup and Sync behavior. * G Suite administrators can still restore items from any emptied trash on behalf of their users for up to 25 days. * Retention policies set by G Suite administrators in Google Vault are not affected by this change. * These changes apply to all G Suite editions and end-users.