New for AWS Lambda – Container Image Support | Hacker News Reader

New for AWS Lambda – Container Image Support

aws.amazon.com

208 pointsappwiz89 comments

Hey everyone, we're really excited about this feature launch, and I wanted to come in to clarify any misconceptions.

With this capability you can now package Lambda functions using familiar container image tools (Dockerfile, cli tools, build systems) but you still need to code them for the event model, have a handler, etc.

It's a big improvement, but its not "run any container in Lambda".

Either way, hope you go and try it out and we'd love feedback on how it can be better.

- Chris Munns, Lead of Dev Advocacy - Serverless@AWS

Hi Chris;

One thing I'd like to see is buildpacks for that final function contract. It's been done before for other cloud-y respond-y things (we did it on Project riff, Google have done it for Cloud Run), so I am aware it's possible. The nice part is that you won't need to build all the buildpacks yourself -- just the small set that adds your specialisations.

Feel free to email me.

Yeah, this is Day 1 for this and I think we've got a bunch of ideas to make it easier in the future. Most importantly we're looking for feedback just like this!

Thanks, - Chris

hi Chris, maybe slightly off-topic but is there a chance to see gravitons and/or ML chips on Lambda? :)

Couldn't share future roadmap here, but these are some cool ideas :)

> but you still need to code them for the event model, have a handler, etc.

>It's a big improvement, but its not "run any container in Lambda".

Well that’s... incredibly disappointing and makes this announcement much less exciting.

The entire point of containerization is portability across different services and platforms. It’s seems like a massive miss for the team to tout “container support” but then still require platform-specific customizations within the container. I pretty much don’t care at all for this as-is.

Are there plans for “run any container on Lambda”? That’s what everyone was (falsely) excited about.

Your reading of "its not 'run any container in Lambda'" may be a bit too pessimistic. From what I'm seeing, you can run any container (<10 GB), but it just has to implement the Lambda Runtime API[1]. You can't run a random container and expect Lambda to know how it should communicate with the world.

As others have noted, ECS or Fargate would be more appropriate for cases that fall outside the Lambda event model.

[1] https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.ht...

> you can run any container (<10 GB), but it just has to implement the Lambda Runtime API[1]

So in other words, you can’t run any container.

Again, the entire point of containers is portability across execution environments. If I have to build special containers specifically for Lambda because they require these special runtimes, that defeats the entire point.

> You can't run a random container and expect Lambda to know how it should communicate with the world.

Google Cloud Run, which everyone keeps comparing this with, works exactly like that. Upload any random container, tell it which port to open, and bam... running application. You don’t have to mess around with adding any “Cloud Run runtimes” or modifying your code to add special “Cloud Run handlers”. Because that would be silly.

It's not really a very good comparison to be honest, because Lambdas integrate with a whole bunch of AWS services that send them events that aren't HTTP requests via a port.

I had exactly the same thought you're expressing here when I first built a Lambda to serve as a HTTP API manually after previously using Azure a tiny bit. In Azure you write their equivalent function and one of the built-in trigger options is HTTP, you enable that and immediately get a URL. Lambdas are quite close to that now I think if you're using the console UI, but you used to have to go through API gateway and set everything up manually.

But the point is that Lambdas aren't HTTP APIs that listen on a single port. They receive events that can be proxied HTTP requests via API Gateway, or messages on queues, or completely arbitrary invocations from step functions, or notifications that look nothing like HTTP from S3 buckets. I'd like the Cloud Run model when my lambda is acting like an HTTP API, but I don't think it'd be much fun to have to treat every AWS event as a full HTTP request for everything else.

I'm not very familiar with Google Cloud, but I think their equivalent is Cloud Functions which looks very similar to Lambda's pre-container model: https://cloud.google.com/functions/docs/writing

I think your integration comment is spot on but might be an apples and oranges thing. On GCP everything is an HTTP request with an id_token in the header and event in the body, almost surprisingly so. As a result services like Cloud Functions and Cloud Run containers are quite literally just generic HTTP handlers. I suspect the integration feel of Lambda, which I agree with, might be a consequence of inter-service communication on AWS being relatively proprietary, or if that's not the right word, custom? Maybe the loose feeling of GCP is because everything is an HTTP request? It's almost like all of GCP runs on a built-in HTTP API Gateway setup by default. Honestly most of the time I feel like I'm living in HTTP-Request-land anyway, so not having to context switch when working with the platform is kind of nice sometimes.

Thanks for mentioning that, it definitely sounds like you can get a lot further with a HTTP handler in Google Cloud than you can in AWS. I can definitely see why avoiding that context switch is nice. I can also imagine it making things like metadata a lot easier if they use common and well documented headers, where in AWS every service will inevitably have its own way of putting stuff into the event. Everything in AWS feeling custom would be a pretty polite and fair way to put it IMO!

Do Google represent a JSON body HTTP event in a clean way? In AWS when a lambda is working as one, it receives the body as a string in the JSON representing the request, which you have to JSON decode. Again because not everything a Lambda can act on is JSON and your API doesn't have to be, what else can it really do I guess. But it does make it a bit horrible to generate that data to test with - we actually have a small script we use locally to convert a nicely formatted JSON file with test data into their HTTP-request-as-json format because it's so annoying to work with a large encoded JSON body. It's a small thing, but it means when you switch to working with a plain AWS event it's one less thing to think about. And I guess even if you're in HTTP-Request-Land and everything is nice JSON, it's still all going to be custom between services from there on by necessity.

You can run any container. It just wouldn't run as you expect if you don't implement the lambda runtime api.

> Google Cloud Run, which everyone keeps comparing this with, works exactly like that.

Everyone does keep comparing this to GCR, wrongly. Not just because Lambda is not "bring any container" whereas GCR is, but more importantly because GCR and Lambda have very different operational models. GCR is a serverless platform for hosting web servers. Lambda is a serverless platform for event handling. The latter is a super-set of the former and thus requires more specific tools.

Yes, theoretically there could be a setup whereby you expose a port, and some lambda intermediary translates the invocation payload to an HTTP message your container can read. But I am endlessly fascinated why anyone would want that. I laugh a bit on how we've hit peak AWS where people here are legitimately asking for an HTTP request to hit an ALB, the ALB translates it into a lambda payload object, then a lambda component re-translates that payload back into an HTTP request, so your application can translate it into an object. Do you understand how insane that is?

The only tactile advantage lambda has over fargate is scale-to-zero. I think saving $7 per month is a pretty bad trade-off for the insane performance overhead and complexity a true docker-in-lambda solution would necessitate, and thus be unattractive to most consumers.

I think you probably want Fargate for that.

This announcement is great if you need to package stuff inside your container for use at Lambda runtime.

+1.

That's the idea. Run any container in Fargate. Now use the same tools you use to build/package those containers in Lambda.

1. Sorry you are disappointed by this. 2. This is why I wanted to post here, to make it really clear. Andy only got to spend a few seconds on this and couldn't get into all the nuances. The launch post does, and we'll have more posts over the next 3 weeks just on this topic.

Containerization solves a few things. One big one was the container image as a packaging model. As customers struggle with dependency management or installing native packages (RPMs) we're basically faced with reinvent Dockerfile... or just use Dockerfile. This is an over simplification of what's happening, but that was the initial spark of this, many many cycles ago.

I think this is still going to be really valuable for folks, but what you are looking for already exists I'd say, and is Fargate.

Btw, ppl shouldn't be downvoting this, its all very valid.

- Chris, Serverless@AWS

> but what you are looking for already exists I'd say, and is Fargate.

No, Fargate isn’t that at all. Google Cloud Run is what you meant to say.

This is of course still valuable to allow container-based workflows to adapt to Lambda, but it really seems like AWS missed the mark on identifying why people really want to use containers. Just one look at the amount of people on HN threads or on Reddit excited at using “arbitrary containers on Lambda” should tell you what people really wanted - and now they have to come away disappointed.

> but what you are looking for already exists I'd say, and is Fargate.

IIUC though, Fargate doesn't have "scale to zero" like Lambda and API Gateway. Then again, IMO, scale to zero and the associated cold starts probably aren't the best fit for handling HTTP requests that are waiting for an answer right now.

Cloud Run on GCP is the “run any container” solution that this isn’t. It scales to zero, responds immediately to an incoming http request and can handle up to 50 concurrent requests out of the one invocation for no additional cost.

What can we expect cold start times to look like with larger containers? (1GB for example)

We optimize and cache the image so cold starts times should be the same as with zip functions.

Wow!

Hm, not what I observed so far. Same Go application taking 1.5s instead of ~50ms to cold start.

It's unclear from the docs -- are the container images supported by CloudFormation?

Yes, either those docs will ship today or sometime this week. They are doing everything in batches but I've seen the SAM support pre-launch (which would require it).

- Chris

Awesome. Very excited about this!

Thanks for coming here to answer questions. Much appreciated!

Does each new Lambda cold start pull the entire image from the repo? Or if I derive from Lambda base images am I likely to get some of the layers already cached?

I'm trying to think about how the data transfer costs are going to each time the Lambda is instantiated. I didn't take this into account with Fargate and got burned when trying to trigger images on demand in a similar way.

This could replace my whole SQS / CloudWatch Events / Fargate setup using Lambda if I can figure out cold start costs ($ not time).

There's a few things at play. Functions will still stay warm inbetween invocations and will keep local any data already in the worker. We also maintain a couple different levels of cache so as to not hit ECR often.

I know we've got a few blog posts coming out over the next couple weeks on this new feature, and each tells a few bits and pieces about the story.

Depending on volume you'll probably find that Lambda will be cheaper for that workload, especially with the new 1ms billing.

- Chris - Serverless@AWS

FWIW, a little experimented I just ran showed me that with simple layers the cold start time of my little 3MB Go app was <100ms, using the Docker image `amazon/aws-lambda-go:1` instead took ~1500ms.

- - - -

REPORT RequestId: f905d5fe-a64e-48c8-b1f2-6535640a6f82 Duration: 7.55 ms Billed Duration: 1309 ms Memory Size: 256 MB Max Memory Used: 49 MB Init Duration: 1301.10 ms

- - - -

REPORT RequestId: 89afb20d-bc49-4d89-91f0-f1ef62ac99aa Duration: 12.20 ms Billed Duration: 13 ms Memory Size: 256 MB Max Memory Used: 35 MB Init Duration: 85.37 ms

Why does the lambda container has to pull events from the lambda runtime using an HTTP GET request, instead of having the runtime push the events with an HTTP POST?

This is all based on how the RunTime API worked already (pre-dates this launch by 2 years or so). We wanted to not change too many bits.

Since function code has no listening socket/port as it were you need something (like the bootstrap script in a custom RunTime) to pull the local interface for the runtime API. That runs on the underlying worker and communicates with the API for Lambda.

How would the runtime know for sure when the container was ready to accept the request? The lambda model--with the container pulling the events instead of them being pushed to it--seems like the correct way to model the concurrency.

Retries and health checks.

Just wanted to say thanks for this new feature. The ability to use images up to 10 GB is huge. Being able to customize the container image down to the base OS is also nice. Don't let the negativity around the Lambda-specific bits get you down. I know that Lambda is about more than just serving HTTP, and I for one plan to use this new feature for a non-HTTP use case soon.

Thanks! Appreciate the feedback and hope this helps you out. Let us know if it doesn't :)

- Chris

I do have a question. I know that Lambda normally reuses a running container for multiple consecutive function invocations. What if I don't want to do that for a particular function? Suppose, for security, I don't want any leftovers from a previous invocation (in case it had data from a different user). Is there a way I could gracefully tell Lambda to create a fresh container instance for each invocation, and just live with the cold start penalty every time? Edit: I could just look to Fargate at this point, but it sounds like Lambda is doing some extra cold start optimization.

No good way to do this today. You are right in that you'd be forcing cold-starts. You could use a Lambda Extension to provide some sort of after-processing clean up of vars or /tmp space.. but thats hypothetical I haven't seen anyone do that yet.

I think what I'll do is write a container entry point that cleans up temporary files like you said, but also repeatedly spawns a new process for the main program, to minimize the findable leftover data in RAM. Just in case an attacker finds a Heartbleed equivalent in my application.

Thanks again!

Just a heads up we're trying to do that with Fargate right now and theres a limit to how quickly fargate can spin up new instances (like max 10 at a time); it's not well designed to do a single-execution-per-message right now, at least at a decent load.

My wishlist for this feature (and for AWS Lambda in general):

- Don't make me build the Docker image myself. I'd like to be able to deploy lambdas by sending you a Dockerfile and having you build it for me, with as few additional steps as possible.

- Make it as easy as possible to run containers that speak HTTP, like Google Cloud Run does. Having a wrapper that translates the AWS Lambda runtime into HTTP calls, maybe even as a proxy I can run inside my own container, would help a lot here.

- Make it as easy as possible for me to host my lambda as the "root" URL of a domain or subdomain that I own - without those nasty /api/ path prefixes.

Zeit Now (the Docker-based predecessor to Vercel) did this the best in my opinion - they had an HTTP API that you could POST a Dockerfile to and they would build it into an image and then start a container listening on an HTTP port, with an HTTPS proxy in front of it, running on a subdomain that I could easily customize.

I would LOVE to see AWS Lambda offer an equivalent developer experience. I'm super-excited about container image support but I'll admit I am absolutely dreading the several hours (to several days) it's going to take me to figure out how to actually get my code running on it.

Awesome, thanks for sharing this.

One of the difficulties with this is that Dockerfiles are usually not very reproducible, mostly due to the design of the distributions they are based on. It could be very surprising to send the same Dockerfile to the endpoint twice and get very different runtime results.

I'd expect that I would send a Dockerfile once and have that compiled into a lambda for me which would then run in perpetuity unless I sent over a fresh Dockerfile to over-write that deployment, so non-reproducible Dockerfiles wouldn't bother me.

I'm OK with developers needing to understand that a Dockerfile may build differently each time if they don't take extra steps to prevent that (like pinning installed versions etc).

This doesn't seem to me like something that should be a core part of the lambda product, but maybe some sort of simplified build pipeline service that has Lambda as a target.

Possibly using AWS CodeCommit + CodePipeline.

CodeBuild builds Docker images. You can write a proxy wrapper in a few lines. Write a lambdalith that processes all the root level functionality.

Does the container support state?

Same as Lambda has before. The state you have is most likely short lived as we reap worker environments every so often, but in-between invokes you could persist some data in memory or tmp space.

I know it’s not necessarily a container related thing, but one I thing that frustrates me a lot, and why deploying takes longer than I expect is because zipping up and uploading a package is a bunch of unnecessary clicks. I just want to “npm install” a few third party libraries in the online IDE. (repl.it has a nice experience here). It’s frustrating to shuffle between my desktop, AWS and back again.

If we push an image with an AWS base image do we need to constantly update and rebuild + redeploy it or do you handle the base image updates behind the scenes (e.g. security patches) like normal lambda?

Would this end up cheaper than fargate spot pricing?

So, apparently I am crazy for thinking that running zip is trivial whereas the ridiculous stack of tooling that surrounds the awkward nested format of docker containers is extremely annoying :/... with the new ability to accept large containers, can I now send you a larger zip, or if I want a larger lambda function am I going to have to use the docker container interface?

This feels like a pretty monumental change - my team(s) have historically avoided Lambda specifically because the local dev flow is such a pain and/or open to interpretation, but local dev tooling for building and working with containers is great. At the same time, creating task definitions for one-off tasks in ECS or similar is also kind of a weird workflow if the task is shortlived or event-driven.

Using an actual container orchestration platform when you need it, and a lambda to run the same container image one-off when you don't need orchestration, feels like a best of both worlds.

I like the idea of "serverless" being more "your packaged execution environment, running with no regard for the underlying host" than "your code, running with no regard for the underlying execution environment".

EDIT: Disregard.... see the AWS team's responses in this topic for the full story, its not quite as convenient as anything packaged via standard container being able to run on Lambda

Worth noting this isn't bring any application and publish as AWS Lambda, but rather build from a specific Lambda base image and package your Lambda code into it. Fargate likely closer to what you are describing here

You don't have to use their base image, you can implement the Lambda API if you want. They're supplying API toolkits for various languages.

You don't need to build from a Lambda specific base image - you can use (nearly) any base image you choose. Your application just needs to implement the Lambda runtime API, which we provide clients for in multiple languages.

Sure - and looks like runtime API isn't terrible, but still not a "lift and shift my express app as a Lambda" which I can do with Cloud Run

This is awesome! Last I looked into it AWS' SAM tooling for running locally actually worked by using third-party docker images that were somehow built by ripping the file system from Lambda, and hooking your code up as a volume (and optionally adding layers as volumes) and then pointing `aws lambda invoke` commands at it. It was all built on https://hub.docker.com/r/lambci/lambda/

Having official base images, being able to think about lambdas as just a docker container in terms of tooling, and losing all the complexity associated with Lambda's custom code deployment logic will be a massive win.

Yes, with the new container image support, you can use the same docker or ECR images to build, and test your functions locally, and run them on Lambda. Consistent images is super useful.

Agreed! Testing AWS Lambda functions that have any complexity to them at all is frustrating. Even using the SAM tools, I found it difficult to test reliably. Specifically, things like environment variables and bundled executables were a pain to get exactly correct. I'm looking forward to using containers in Lambda functions and improving the tooling.

This solves two big pain points for me:

1) the 250 mb size limit.

2) trying to get some tricky dependencies into a lambda compatible image.

Ive got one build system that was going to have to move off lambda in the next few months because of the size issue. We’re shipping a slowly changing node modules directory to lambda, and then repeatedly building an app with them.

We’re also doing perf testing with lighthouse, and that was a pain and a half to get packaged. This should make it far simpler.

How hard would it be to write a tool that can run inside one of these containers responding to the AWS Lambda runtime API interface and proxying it to a regular HTTP server running on a port inside the container?

Such a tool would mean I could take any existing HTTP container and turn it into an AWS Lambda Container Image just by adding the proxy process that translates between the lambda runtime and the internal HTTP server.

That would be like a ten line bash script (the AWS documentation on custom runtimes is actually just a shell script that uses curl to show how easy this is).

Was already working through porting my startup’s web app from docker to lambda via layers and such, but it was a hassle. Then I read today about the 1ms billing and got motivated. THEN I see this about running containers, my life just got much easier. What a time to be alive!

There's going to be a lot more information coming about this over the next 3 weeks during re:Invent. A good outside-AWS blog post is this one: https://twitter.com/hichaelmart/status/1333837825222078466

This is pretty cool, to me it looks like it's moving towards Google Cloud Run's feature set.

One small note - Cloud Run is KNative as a service. Lambda is divorced from Kubernetes.

Personal opinion - I prefer Lambda over KNative. KNative feels like more of a shoehorn into Kubernetes than most other projects. Lambda feels more mature.

Still glad to see this move, regardless.

How so? With Google Cloud Run, you don't really interact with Knative or Kubernetes, so the underlying platform is a non-issue.

There are two versions of Cloud Run. The fully-managed cloud-only variant is actually an API skin over Google App Engine. The "for Anthos" (née "for GKE") variant is Knative[0] in both API and sourcecode.

AWS could plausibly do exactly the same thing: provide a skin over Lambda that speaks Knative.

I think Amazon's and Microsoft's reluctance to wade into Knative has been wariness about Google's intentions. And maybe 6 months ago I would have said "fair enough, I would be cautious too". But the recent update to Knative's governance model largely resolves the problem, in my view. I would very much like to see AWS and MS involvement.

[0] "Knative" is the correct spelling, FWIW.

Disclosure: I work for VMware, which participates in Knative development.

I also, like, wrote a book about it.

It's pretty embarrassing for HN that this would be reflexively downvoted. It's 100% accurate.

That was my initial thought too, but looks like these lambdas have to derive from a specific set of Lambda base images. Cloud Run lets your publish ANY docker container from any base image. So this feels much more like a deployment and development feature of Lambdas and not "run any container as a Lambda"

> looks like these lambdas have to derive from a specific set of Lambda base images

About that, you don't need to use their base images, "You can deploy your own arbitrary base images to Lambda"

Not entirely. This is something we've been thinking about for years now and is really about tooling and developer workflow. That is where the biggest benefit is for this.

- Chris - Serverless@AWS

Why doesn’t AWS have a google cloud like offering?

What would really open this up to new use cases now would be to drop the 15 minute timeout. I've got jobs I had planned to run in Fargate that would really be more appropriate for Lambda, but in some cases I need them to run as long as 90 minutes.

This is great. I recently went through the exercise of deploying a periodic lambda function using AWS SAM. The development workflow was great, except when I needed binary dependencies for the MSSQL database driver. Learning and using Lambda layers was cumbersome and the bulk of the dev time. This change will make the lives of engineers in docker shops much easier.

This should make things easier if you're mature in container tooling. AWS SAM recently added much better support for Lambda Layers, and you can use makefiles etc. there. In fact, I would say building layers with SAM is probably the easiest way to build layers! Now you get to try with container images.

Does this require API Gateway or can you run it directly via Application Load Balancers like regular Lambda?

There's no impact of this feature to how you invoke your Lambda functions.

Ah great thanks.

One more q: What about network latency on warm up when connecting to e.g. RDS?

This is fantastic news.

I'm very curious about the invocation latency overhead of Docker. I assume its non-zero relative to the more native runtimes?

OCI in general doesn't need to have very much overhead at all; it's just a few syscalls after fork() and before exec() (that trivializes it a bit, of course). I wouldn't be surprised if native runtimes already do some of the same work; it's just good practice when designing mixed-trust systems on Linux.

Docker's specific implementation may offer other hurdles, of course, but AWS Lambda could easily use one of the many other implementations of OCI.

Hi, I work in the AWS Serverless Team. Just to say, the performance of running a container image is pretty much the same as a function packaged as a zip function. We cache the container images near where the function runs so startup time isn't any worse than ZIP.

After doing some K8s stuff in the past months, I'd rather prefer the direct ZIP upload to indirections with a registry.

Currently it's only ECR, and public Docker hub. I hope they will support private registries soon.

So this is google cloud run but in aws? Always been a huge fan of cloud run.

Rust base image?

Is there some easy way to simulate api gateway + docker lambda locally?

Hi, I work for AWS in the Serverless team. Yes you can run Api Gateway and Lambda locally for testing. Have a look at AWS SAM. https://github.com/aws/serverless-application-model

How does this compare to fly.io?

Fly.io is more like a simplified Fargate.

This is more like Lambda with Docker instead of ZIP. More code storage but same 15min runtime limit.

Reply on news.ycombinator.com