Hacker News Reader

Starofallon Show HN: S3HyperSync – Faster S3 sync tool – iterating with up to 100k files/s

For a big sync this is definitly the way to go. But for a simple daily sync we found the tool more usefull. Especially when syncing to a non AWS S3 provider that does not offer a inventory feature.

Starofallon Show HN: S3HyperSync – Faster S3 sync tool – iterating with up to 100k files/s

File size is easy to compare, so you at least know that the full file got sycned. Hashes is a bigger issue with AWS in general, as you only get an ETag from S3 which is the MD5 of a file for Puts, but not for multiparts - and also dependent on the multipart size. So you cant really check the file for hash equality easily.

The good news is that with S3 over HTTP you should not really run into byte flip issues.

The sync server does not need any file system storage, it processes all uploads in memory and only ever buffers 5MB per worker for multipart uploads.

rclone looks like a good alternative, but without the focus on fast iterations for e.g. daily backups of huge buckets.

Starofallon Show HN: S3HyperSync – Faster S3 sync tool – iterating with up to 100k files/s

Profile support might also make things less easy to use, as the tool has support for cross S3 provider syncing. Thats why it needs the key and secret of source and target seperatly.

Local Sync is on the idea list, but not that simple - as local folders do not have the same "paginate all items in lexicographic order as it would look like on S3" feature ^^

Starofallon Show HN: S3HyperSync – Faster S3 sync tool – iterating with up to 100k files/s

I have not yet compared it against this tool, but the given numbers are for download and not for syncing files.

For a S3->S3 sync using an C6gn.8xlarge instance, I got up to 800MB/sec using 64 workers, but the files were only in average around 50MB. And the bigger the file the higher the MB/sec.

Also from my short look into it, s5cmd does not support syncing between S3 providers (S3->CloudFlare).

Starofallon Show HN: S3HyperSync – Faster S3 sync tool – iterating with up to 100k files/s

The code is really extremly simple, just these few files: https://github.com/Starofall/S3HyperSync/tree/main/src/main/...

1) The underlying S3 Framework is already super fast https://pekko.apache.org/docs/pekko-connectors/current/s3.ht...

2) Lots of multithreading, stream buffering and pipelineing

3) For the fast iteration speed the "read,parse,ask for next" loop is the main bottleneck - so if you e.g. know that your sync source prefix contains uuids - the tool creates a file iterator for each known subfolder prefix. And with 16 iterators, its mainly the CPU that bottlenecks the XML parsing :)

Starofallon Show HN: S3HyperSync – Faster S3 sync tool – iterating with up to 100k files/s

Yes! That was one usecase we had, that the AWS CLI was not able to do. You can sync any two S3 compatible services with each other, and it even supports legacy path-style if you e.g. want to sync to a local Minio S3 installation.