Hacker News Reader

Comment by niklasrde | Hacker News Reader

We have recently set up a TensorFlow assessment function in AWS lambda, and got very close to the maximum allowed size of a lambda function (250MB) with the trained model currently being 85MB, and the TensorFlow libraries and binaries taking up another 140 or so megabytes by default (<- which we cracked down a bit, but that's quite a hack).

I feel like Amazon could do some work in this area to support users to use their own engines and not be bound to AWS AI Platforms and Services.

This could be as simple as publicly documenting the time lambda's stay 'warm' for and retain data in /tmp persisting through multiple invocations or some other examples on how an AI workflow could be implemented with popular custom engines such as TensorFlow.

Does anybody else have any experience in this regard?

View on HN

I've hacked around lambda quite a bit (I think the compressed size of one function is a tad under the max allowed). My hacks I remember are:

- Run strip all .so libraries -- many aren't stripped fully

- In Python I manually deleted sub packages of numpy/scipy I didn't need

- If you're loading large models at initialize, numpy load routines are _much_ faster than cPickle. Have it load at module initialization, not during each invocation.

I should really write a blog post about my experience with it.

At a certain point I decided I was doing something that lambda really wasn't designed for -- I'm looking at migrating off, but the current implementation makes capacity planning super easy. Provisioning 1000 machines with 1GB of RAM for 15 minutes every day to read off a queue isn't a trivial problem.

(Also, if anyone from AWS is reading, being able to limit the max concurrency of a single function vs account level limits would be super useful).

Would love to see that blog post!

That sounds quite similar to what we're doing!

Would love to see a blog post to compare experiences..

(Speaking of blogposts.. that was somewhere on the todo list..)

Just to throw this out there as another possible optimization, if you find that you're putting a big fat library into every function, one possibility is to run the library as its own lambda function. You'll be slowed down a bit by the network but it might be made up for by not having to constantly initialize the same thing.

It sounds like you're stretching to make the use case fit inside a Lambda.

Have you considered extracting the assessment functionality into an SWF activity, invoked from your Lambda via StepFunctions?

No, we haven't looked at SWF at all - might do some reading up on that.

We considered running it on EC2, but the economics just didn't work out for our needs (hundreds of parallel processed jobs with irregular spikes, <5s runtimes per invocation and some others)

You might also consider AWS batch. It's designed for this type of use case.

The keep warm time isn't static AFAIK but some frameworks like Zappa have chosen 5 minutes (I work for AWS). If you raise an issue with support there may be other ways around this. There are also other reasons for not relying on functions being warm (spike in concurrent invocations, AZ outages, latency, etc.). If you open a support ticket and email me the # I'll see what I can find out: randhunt at amazon dot com

I'm curious what sort of things you did to shrink your model?

Have you considered pulling the model data from S3 outside of the main lambda handler and seeing if that negatively impacts performance -- with a cloud watch event running every 5 minutes or so to keep the function warm?

Hey, thanks for your reply.

Hadn't thought about a CloudWatch even to keep the function warm, I might suggest that to the team.

We haven't shrunk the model, we've deleted superfluous files from the TensorFlow python library and dependencies (we don't need tensorboard, for example).

It would be nice if you could package TensorFlow up into a minimal component just for assessment and not have any of the 'learning' stuff or other added-on libraries, but we couldn't find a simple way of doing that - we're not pro C++ engineers and even our python is not the greatest. We're managing for now, but if our model grows any bigger we will run into issues, but there have been some good suggestions in here.

We had considered the S3 store, but we ruled that out quite quickly for cost & performance at the number of invocations a months we're looking at - but that was before we knew more about the 'keeping warm', so that may be revisited, too.

We get around this by using a dependency injection framework (e.g. Guice) and initialising the module statically, so it will only get called once when the JVM loads (i.e. when the Lambda 'warms up') and then inject the dependencies into anything that uses them

For example in one Lambda function, we use Guice to get resources from S3.

Just curious, is this way better than using Cloud ML? https://cloud.google.com/ml/

We're (sadly) a bit locked into using AWS. We've been eying the Google offerings for a while though (not just for this), so who knows what we may do in the future.

Have you tried using Algorithmia for this? We get around a lot of these limitations that Lambda has (you can also call us from lambda and we can host most of your DL models https://algorithmia.com/developers/algorithm-development/mod...).

Disclaimer: work @ algorithmia

Knowing nothing about assement and tensor flow: Could the trained model be downloaded via s3 at init instead of being included inside the function zip? I believe the memory limits are more lax than the actual function limits. And the /tmp limit is 512MB.

checkout this blog on running predictions on Lambda with MXNet. It has pre-packaged numpy, openCV and MXNet.

https://aws.amazon.com/blogs/compute/seamlessly-scale-predic...

I have wondered about use cases for tensorflow - can you give me an example of a simple one ?

Reply on news.ycombinator.com