Hacker News Reader

I have a task:

URLs to 10M images.

I need to create a thumbnail for each of them.

I.e. I need to download each image, resize and save in the webp format to the S3.

Right now I am doing this locally and it takes forever, as my images are in high resolution their total size is a few Tb.

Is there a tutorial that will describe how to do it? I was thinking about using AWS Lambda + SQS, but if DStack is an easier solution, I would be happy to check it out.

View on HN

Thank you for the comment! Exactly, this is one of theses tasks, dstack is designed to help with. You basically write a simple Python script that does the job, tests that it works locally, and then run via dstack – asking for a more heavy machine.

The entire setup will take roughly 5-10 mins. Promise to add a tutorial with exact steps and code snippets soon.

In the future, we also plan to support distributed workflows. Actually, this is already supported by the design, we just need to implement a provider!

Reply on news.ycombinator.com