HNHacker News
TopNewBestAskShowJobs

emasquil

135 karma · joined April 10, 2021

submissionscomments
emasquil··on Downloading files from S3 with multithreading and Boto3
Sorry about that! Next time I work on my blog I'll take that into account and find a better theme/color palette. In the meantime you can change it to light mode (clicking on the top right button) and there you have black text on white background.
emasquil··on Downloading files from S3 with multithreading and Boto3
I just mentioned the issue because a colleague of mine did some quick tests with aioboto3 and he told me it was downloading the files sequentially. We thought it might be related to that.

You made a great point about going async if you need a working command channel.

emasquil··on Downloading files from S3 with multithreading and Boto3
Unfortunately I haven't. If someone wants to benchmark this solution I'd be glad to hear about that!
emasquil··on Downloading files from S3 with multithreading and Boto3
In this case I needed to download about 1k files of size ~ 10mb. Doing this with multithreading was so much faster than downloading all the files sequentially.
emasquil··on Downloading files from S3 with multithreading and Boto3
In this case I just wanted to keep it simple and do it the "native" way. I've been hearing a lot about aioboto3 and I'll surely check it out. However, it seems to have some limitations like https://github.com/boto/botocore/issues/458