Hacker News Reader

darthShadowon Comparing ClickHouse to PostgreSQL and TimescaleDB for time-series data

Does the disk usage go down later once the numerous parts are merged or not?

I would assume it does but reading the article implies that it does not.

darthShadowon Downloading files from S3 with multithreading and Boto3

RClone does this and much more.

RClone: https://rclone.org/ S3 Backend: https://rclone.org/s3/

darthShadowon Downloading files from S3 with multithreading and Boto3

RClone: https://rclone.org/s3/

darthShadowon Downloading files from S3 with multithreading and Boto3

RClone does all that you require and much more.

RClone: https://rclone.org/ S3 Backend: https://rclone.org/s3/

darthShadowon Moving 25TB data from one S3 bucket to another took 7 engineers and 2 full days

I did get an update from Google recently that they are implementing automatic clearing of trash after 30 days which may help you.

The exact wording of the relevant section of the mail:

We are writing to let you know that starting October 13, 2020, Google Drive is making a change so that its trash behaves more consistently with the rest of our G Suite services with regards to automatic deletion. This means that any file that is put into Google Drive’s ‘My Drive’ trash will be automatically deleted after 30 days. Items in trash will still continue to consume quota.

Please note that starting October 13, 2020 any files already in a user’s trash will remain there for 30 days. After the 30-day-period files that have been in the trash for longer than 30 days will begin to be automatically deleted.

What does this mean for my organization?

Any file that has been in the trash for longer than 30 days after October 13, 2020 will be automatically deleted. We will be showing in-app messaging in Drive starting September 15, 2020 and in our Editors products (such as Google Docs and Google Forms) starting September 29, 2020.

A few things to note:

* Files in shared drives trash are already automatically deleted after 30 days. * These changes affect items that are trashed from any device and any platform. * Files deleted from Drive File Stream will be purged from the system trash after 30 days. There is no impact to Backup and Sync behavior. * G Suite administrators can still restore items from any emptied trash on behalf of their users for up to 25 days. * Retention policies set by G Suite administrators in Google Vault are not affected by this change. * These changes apply to all G Suite editions and end-users.

darthShadowon Moving 25TB data from one S3 bucket to another took 7 engineers and 2 full days

Yep, as mentioned in the other comment, with configurable `--transfers` & `--checkers`, the only bottleneck is the amount of CPU you have available and can allocate.

darthShadowon Moving 25TB data from one S3 bucket to another took 7 engineers and 2 full days

RClone. Probably the best tool I have come across for interacting with cloud storage solutions.

Move Docs: https://rclone.org/commands/rclone_move/

S3 Docs: https://rclone.org/s3/#amazon-s3

Supports parallel server-side copies and deletes (no server-side moves, unfortunately) so this would have been much faster.