Hacker News Reader

The source bucket had a ton of small files and most of the time was spent making network requests (O(<number of files>)). The suggested solution was to do a batch operation (open a ticket for a batch operation). Would rclone have helped in this scenario?

View on HN

Yep, as mentioned in the other comment, with configurable `--transfers` & `--checkers`, the only bottleneck is the amount of CPU you have available and can allocate.

> the only bottleneck is the amount of CPU you have available and can allocate.

In this case S3 will be the bottleneck. 25 TB will have around 700M files (assuming it's as dense a linux installation). S3 can do 5000 operations per second. It'll take at least 40 hrs (assuming everything works at peak speed during the whole process).

rclone lets you specify how many files to handle in parallel. It defaults to 4, and auto retries in the event of failures.

Ramp up the parallelism, and don’t make 7 people monitor it round the clock.

Reply on news.ycombinator.com