S5cmd: Parallel S3 and local filesystem execution tool
github.com
github.com
My only headache was that I was invoking it from python, and it does not have bindings, so I had to write a custom wrapper to call out to it. I am not sure of the difficulty of adding native support for Python, but I assume its not worth the squeeze and just calling out to a subprocess will work for most user's needs.
https://raw.githubusercontent.com/peak/s5cmd/master/doc/benc...
I've implemented at work once a rudimentary parallel uploading of many small files to S3 in Python and with boto3 (was not allowed to use a third party library or tool at that time) because it's soooo slow to upload many small files to S3. It really takes ages and even if you just upload 8 small files in parallel it makes a huge difference.
ssh user@rsync.net s5cmd
Password:
NAME:
s5cmd - Blazing fast S3 and local filesystem execution tool
USAGE:
s5cmd [global options] command [command options] [arguments...]
> Backup Tools
> attic, borg (usage notes), rclone, unison, s5cmd and git are installed (on our server side).
> You would most likely call these commands by running these tools locally and connecting to rsync.net with them.
I'm surprised by these claims. I have worked pretty intimately with S3 for almost 10 years now, developed high performance tools to retrieve data from it, as well as used dedicated third party tools for performant file download tailored for S3.
My experience is that individual S3 connections are capped over the board at ~80MB/s, and the throughput of 1 file is capped at 1.6GB/s (at least per ec2 instance). At least I have never managed myself nor seen any tool capable of going beyond that.
My understanding is then that this benchmark's claims of 4.3GB/s are across multiple files, but then it would be rather meaningless, as it's free concurrency basically.
Downloading that 10GB file to /dev/shm with s5cmd took 24s, all while spawning 20 or so threads which were all idling for IO.
The same test using a Python tool (http://github.com/NewbiZ/s3pd) with the same amount of workers took 10s.
Cranking up the worker count of the latter library until there is no more speedup, I can reach 6s with 80 workers. That is, 10/6 = 1.6GB/s, which seems to confirm my previous comment.
What am I doing wrong ?
The claimed numbers are _not_ reached with S3, but rather from a custom server emulating the S3 API, hosted on the client machine.
I think this is very misleading, since these benchmark numbers are not reachable in any real life scenario. It also shows that there is very little point in using s5cmd compared to other tools, since beyond 1.6GB/s the throttling will be from S3, not from the client, so any tool able to saturate 1.6GB/s will be enough.
I wonder what the limit is for uploads inside AWS. I've never been able to afford EC2 instances with the requisite networking or I/O performance.
You can spawn multiple connections to S3 to retrieve chunks of a file in parallel, but each of these connections is capped at 80MB/s, and the whole of these connections, while operating on a single file, to a single EC2 instance, is capped at 1.6GB/s.
Also you should up your chunk size to limit tps
Source: running s5cmd in prod for 1yr+
It supports glob patterns like so, and will do smart filtering at every stage possible: */2025-0[45]-*/user*/*/object.txt
I haven't done real benchmarks, but it's parallel enough to hit s3 parallel request limits/file system open file limits when downloading.*
For s3 mounts I would use geesefs.
Have to later take a look at s5cmd as well...