Hacker News Reader

Comment by ramoz | Hacker News Reader

Genuine question... Does it support the size of a typical (for data, ml, image proc, etc) Python dependency bundle?

The deployment package size limit seems to be 250MB: https://docs.aws.amazon.com/lambda/latest/dg/limits.html

That's the limitation for data related worker tasks. The iron.io can be alternative (they offer 512 MB https://try.iron.io/pricing-worker-monthly/)

There are workarounds for that, f.e. on startup loading code from an s3 bucket (https://hackernoon.com/exploring-the-aws-lambda-deployment-l...)

That workaround is for the 50MB limit on zip files, not for the 250MB limit :)

No, you get another 500 MB of tmp space you can use at runtime. This workaround does work.

That would be OK for some ML applications if the framework itself would not be included in the deployment package.

I have a deployed lambda with your usual stack ( numpy sklearn scipy)

Did you manage to do that without pulling them from S3 (i.e., get them to fit in the 250MB limit)?

I'm using serverless-python-requirements (see the specific example https://github.com/UnitedIncome/serverless-python-requiremen...) - it uploads the libs zipped and unzips them on first invocation in the handler.

For a lot of data processing and indigestion in a commercial setting, the standard library is often enough.

Had a Lambda that parsed a file attached to an incoming email, and it worked very well. Even with complex packages like lxml, it usually isn't an issue. Seems like it might be tricky with some scientific packages or ML stuff though, but Lambdas aren't great for those workloads anyway, although the 15 minute runtime increase has helped.

Reply on news.ycombinator.com