AWS Lambda Supports Python 3.7 | Hacker News Reader

AWS Lambda Supports Python 3.7

aws.amazon.com

111 pointspplonski8620 comments

Honest question, why does it matter what version of python lambda supports (or any framework / language for that matter)?

Couldn’t you just pick the go runtime for example and have it execute the framework / language you want with minimal overhead given it will be a call between two processes running on same host, which you can keep running between invocations.

AWS just released a solution to run any programming language on AWS Lambda: https://aws.amazon.com/blogs/aws/new-for-aws-lambda-use-any-...

Ok, so I take from this that the post about specifically Python 3.7 doesn't make much sense really (as a HN post).

AWS provides the Python 3.7 runtime for you, so you don't need to bundle it. The post refers to runtimes that AWS doesn't provide, such as Haskell.

It makes sense because this is an AWS supplied executable, where you just upload the .py files.

If you follow the "Run any language" approach, you need to bundle the python interpreter and any libraries required.

> why does it matter what version of python lambda supports (or any framework / language for that matter)?

Python 3.7 is a lot faster (30 - 40%) than Python 3.6 for certain types of functions.

And you can't (easily) just install Python 3.7 from another runtime, because it has a ton of dependencies and takes minutes to compile. So even if it's technically possible, it precludes a lot of use cases.

I actually dealt with a similar problem a couple years ago, we were trying to get numpy on Lambda but there's C compilation that needs to happen. What I ended up doing was provisioning an EC2 instance with the same settings as the environment that Lambda uses, then building everything there, and shoving the binaries in the Lambda. It worked, I'm pretty sure the same thing would work for building Python.

Genuine question... Does it support the size of a typical (for data, ml, image proc, etc) Python dependency bundle?

The deployment package size limit seems to be 250MB: https://docs.aws.amazon.com/lambda/latest/dg/limits.html

That's the limitation for data related worker tasks. The iron.io can be alternative (they offer 512 MB https://try.iron.io/pricing-worker-monthly/)

There are workarounds for that, f.e. on startup loading code from an s3 bucket (https://hackernoon.com/exploring-the-aws-lambda-deployment-l...)

That workaround is for the 50MB limit on zip files, not for the 250MB limit :)

No, you get another 500 MB of tmp space you can use at runtime. This workaround does work.

That would be OK for some ML applications if the framework itself would not be included in the deployment package.

I have a deployed lambda with your usual stack ( numpy sklearn scipy)

Did you manage to do that without pulling them from S3 (i.e., get them to fit in the 250MB limit)?

I'm using serverless-python-requirements (see the specific example https://github.com/UnitedIncome/serverless-python-requiremen...) - it uploads the libs zipped and unzips them on first invocation in the handler.

For a lot of data processing and indigestion in a commercial setting, the standard library is often enough.

Had a Lambda that parsed a file attached to an incoming email, and it worked very well. Even with complex packages like lxml, it usually isn't an issue. Seems like it might be tricky with some scientific packages or ML stuff though, but Lambdas aren't great for those workloads anyway, although the 15 minute runtime increase has helped.

Elastic Beanstalk meanwhile is still on Python 3.4

For the preconfigured Docker version, yeah. But for the regular Python Beanstalk they released 3.6 at roughly the same time that Lambda released 3.6 support, so maybe there is some hope there.

Reply on news.ycombinator.com