On a related topic, I've had a ton of problems finding a cloud storage system that will reliably handle files around 100-200gb. Does anyone have a recommendation for a service that can handle that file size with ease?
S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.... (Max object size: 5TB, Max single multipart size: 5GB)
Backblaze B2: https://www.backblaze.com/b2/docs/large_files.html (Max file size: 10TB, Max single object size: 5GB)
Azure: https://docs.microsoft.com/en-us/rest/api/storageservices/un... (Max file size: 4.75 TB, Max single block size: 100MB)
Google: https://cloud.google.com/storage/quotas (Max file size: 5TB, doesn't appear there is a lower limit for objects to be composed into a single object, docs could be better in this regard)
@khc: Terminology updated to be more clear for S3
It's reliable enough, if you can get it to Microsoft's cloud. But for the last six months I've struggled putting very large files into Azure, using five different connections from five different providers in three locations. Small files are no problem. But large ones take two, three, or four tries.
You can compose a 4TB object with a 1 byte object, or you can compose 32 150GB objects, just so long as the destination object doesn't go over 5 terabytes.
rsync.net gives you an empty, ZFS filesystem that you can do anything you like with.[1]
I believe the file-size limit is 16 exbibytes (264 bytes).
rsync.net can also talk to every cloud storage provider[2] because rclone[3] is built into the platform:
ssh user@rsync.net rclone s3:/some/bucket rsync/home/dir
2**64