A shared file system for lambda functions
aws.amazon.com
aws.amazon.com
I tried Lambda for a use-case that I had in 2018:
We published Polls and Predictions to people watching the 2018 World Cup. We set the vote callback URL to a function on AWS Lambda.
It failed spectacularly during our load-testing because the ramp-up period was far too slow. We needed to go from 0 to 100,000 incoming requests/second in about 20 seconds.
We had to switch to an Nginx/Lua/Redis endpoint because Lambda was just completely unusable. It would have cost us $27,000/month to pre-provision 10,000 concurrent executions...
What is it that people actually use Lambda for?
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/S...
1) AWS administration-related actions (required for non-trivial mgmt.)
2) end-user actions (optional).
AWS throttles and limits everything, so expecting 100,000 requests in 20 seconds on a default AWS account is unreasonable.
The Cloud doesn't mean unlimited capacity.
Also, I haven't seen anybody version control lambda code, even in compliance environments, so something to investigate.
Edit: my AWS support rep explained it: Without pre-provisioned capacity, it would take 34 minutes to scale our Lambda functions to the peak concurrency we needed. And it would cost several thousand dollars.
Completely insane cost. Also, EFS is ridiculously expensive.
You would need to pre-arrange that with your account manager too.
An ALB is hosted on servers, and when you hog those servers, they would need to migrate to bigger.
We have anything that gets deployed to lambda go through a jenkins job. Pulls code from GH and builds/uploads it to lambda. Very few people have the ability to manually edit lambda jobs through AWS. While it isn't a perfect solution, it's worked pretty well for us.
What ? That does not make any sense to me. I know I do and I think many do too.
Huh, how is it any different building and deploying to anything else.
I store the code in git, build it in team city, and deploy it to S3 using octopus deploy which updates the lambda to point to the new versioned off zip.
I've built a lot of lambda apps and never used it, pretty much everything I do requires dependencies.
Sure, I used a lambda to turn on/off a build server during particular hours. I wrote it directly into the console.
For actual applications tho, I don't use the console.
Just because the console exists doesn't mean people are flat out not versioning their code.
For us the perfect use case is the production of PDF reports: they aren't run that often; they can be emailed a few minutes later; and when too many of them are run at the same time, they would steal all the RAM on the server.
The perfect use case we see repeatedly is for high volume HTTP requests to API Gateway endpoints that trigger Lambda functions that respond in less than a second depending on what compute is running.
Running Lambda as a back-end is one of the most expensive options (especially when high load comes in, as a parent pointed out). So no question, that's what they want to sell.
Edit 5 months cos it went live in jan.
It is a very common pattern for IoT applications also and well-suited for it. The workloads are often bursty and unpredictable.
But for many things, I believe it is still preferable to stick with a traditional web app or API. It is still early days for serverless and there is a lot of hype around it. As with most hyped things, people see it as a silver bullet and don't think through the tradeoffs they are making.
There are very many users of Lambda that run very high volume production workloads (billions of requests a month) and save an enormous amount of money doing so compared to traditional techniques. A lot of that saving comes from the ability to create solutions in days as opposed to weeks or even months and the fact that for most use cases the capacity exists to handle the volume most users need.
Of course, in this case I have to use it, because Firebase Hosting is static, so this is the only way to handle dynamic urls if the simple rules of firebase.json are not enough.
From direct exposition (via API gateway) to HTTP traffic, to very spiky loads such that we don't pay for hardware in between spikes. We've also have a kind of "hybrid pipeline" that is a mix of both HTTP requests from clients and SQS messages. The HTTP part sends messages to SQS, which triggers lambdas asynchronously so we get retries for free.
Your load doesn't strike me as being out of the ordinary, I suspect you had one of the following: - Slow start, which means AWS was unable to re-use an already warm container to run the function a second start. This is particularly true with Java, although they've massively improved it recently. - A dependency on another service which would have caused the slow start (loading a config is a common unexpected culprit) - You mention 2018, so at that time running a lambda within a VPC was notoriously slow to start - You had a limit on your AWS account (default is 1k concurrent I believe)
1: https://aws.amazon.com/about-aws/whats-new/2020/04/amazon-el...
https://cloud.netapp.com/cloud-volumes-service-for-aws
Whether amazon will ever let you mount it to lambda is a very different discussion.
Do you know how much it costs to write 3 MB/second to EFS? How much does it cost to read 26 MB/second from the same EFS volume?
That's $1,400/month.
I recall Joyent's solution to this (similar) problem where you have an object stored somewhere (e.g. S3) and you want to use that object in a container, but you have to copy it over HTTP or something to do any work on it and the object could be very large.
With Joyent's Manta[1] you would spin up a container right where an object is stored (instead of bringing the objects to the container via NFS.) Also has map reduce support.
This feature they did release deserves little fanfare.
You can just mmap or read the file without doing anything else. That is zero or one memcpy overhead.
S3 clients have to copy the data over the network, assemble the tcp packets, decrypt and checksum for ssl, and then memcpy the result. That’s at a minimum. They may be doing other work, like verifying the s3 checksum, or allocating memory to store the object.
They have to do that once per lambda process, again, at a minimum. They might do it once per lambda invocation.
I wonder how amazon bills DRAM if multiple lambdas mmap the same thing read only.
We were just about to put the script in Fargate (Serverless Docker) and run a ECS task as part of the state machine. Now we don’t have to.
Depending on the use case it can be a good option (some are listed in the article). If the dev doesn't get the disadvantages the rest of the architecture will hardly be correct anyway
As you can seek() on NFS, you can quickly scroll to the middle of a massive file. Unlike S3 where you have to download the whole file first.
One thing that tripped us up at the time was EFS not supporting encryption in transit: but this was fixed in early 2018 when EFS began supporting using stunnel to wrap the underlying NFS connection in TLS. https://docs.aws.amazon.com/efs/latest/ug/encryption-in-tran...
It reads as if this lambda integrated EFS works out of the box with encryption in transit
[1]: https://blog.encrypt.me/2013/11/05/ssl-added-and-removed-her...
To put it mildly, Amazon has a lot more folks a lot smarter than me thinking about these tradeoffs, so I assume they had good reason to think it was fine the first time around. I'm just surprised that the state-of-the-art for cloud-hosted services doesn't presume building on TLS from day 1, so I'd love to know what those reasons are.
It wasn't obvious to me if this is somehow mounting EFS over !NFS (since never says the words NFS in the post). My main fear when people say "Should I use Lambda / Google Cloud Functions / Cloud Run against my NFS server" my response isn't "How would you set that up" it's "Be careful. Cleaning up NFS locks held by clients that have gone away is fairly painful, and you have none of the mechanisms to make sure it exits properly".
Alternatively, you can mount without locking, and then you get one of the comments downthread about "and now you've given functions shared mutable state but with bad primitives".
tl;dr: Cool! ... But, how does this handle NFS locking?
This is NFS and locking is fully supported (the blog includes an example). Because EFS implements the NFS 4.0/4.1 protocols, locks are lease-based and there isn't a need to clean them up. In the unlikely event of a client crash where the client still held locks, they will automatically expire once the associated lease expires.
Is anybody using Lambda to run huge MapReduce jobs? Do people still use Hadoop?
Doesn't this basically just let you have something like HDFS for running large distributed computations with some shared state, without having to reach for S3 or redis?
If your use case can deal with one EFS volume per isolation boundary, you can use IAM to control who can mount what volume, which might be easier to reason about.
Cloud-y DLP tools don't know about EFS.
The EFS/Lambda integration uses EFS Access Points, which allow you to enforce a specific POSIX identity and directory for NFS operations. You can also use IAM policies to require that specific IAM roles/users use a specific access point.
edit: ah, EFS access points are general and you can mount them from EC2 instances ? MUCH better.
With infrastructure-as-code tools, this can be a little clearer. At Pulumi we wrote a post earlier today on configuring the infrastructure needed to use EFS with Lambda, and it boils down to just a few concepts and a couple dozen lines of infrastructure code.
https://www.pulumi.com/blog/aws-lambda-efs/
Some of the complexity here also comes from the fact that EFS is a general purpose managed NFS service, instead of a fully-abstracted Lambda-specific feature. That does add a little additional up-front complexity, but means you can use EFS across all sorts of different compute in AWS - not just Lambda.
More seriously, this is huge. Unix pipes over shared nfs has always been my big data platform of choice (since before the cloud, or even google map reduce). Things finally came full circle.
So now lambda functions can mount persistent block storage.
Next up: allow your lamda functions to run for longer
Then: allow multiple lamda functions to execute concurrently, and indefinitely, as a group, while having block storage mounted
And finally: use your EC2 instances as lambda functions
I also don't think the time limit on lambdas will extend much further than it is today. Not sure on that one.
Is that a bad thing in your opinion? Examples like that were very much a Day 1 use case, or at least intent, for a lot of AWS Lambda. When we were working on CloudFront Lambda@Edge it was explicitly the intent to move the compute out of the traditional data center or EC2 instance. We wanted to enable customers to run their entire application in this fully managed 'serverless' aspect where AWS can optimize along multiple axes on their behalf.
Internally Amazon is a land of APIs, RPC, and federated services/business logic. Getting to the point where each API action, or internal method, would be hosted as a Lambda was entirely a goal.
Source: Principal at AWS. Not in this space directly anymore, but spent time working on CloudFront & Lambda@Edge.
Is it stupid? Yes, but advertising is generally geared toward human stupidity. More specifically, it's geared toward pointy-haired bosses who are being presented AWS Lambda™ who are thinking to themselves "well I heard a lot about lambdas, are these what my team meant?"
[0] - https://aws.amazon.com/blogs/compute/announcing-improved-vpc...
> AWS Lambda functions can now mount an Elastic File System
from the opening paragraph.
What’s wrong with “lambda functions”?
A lambda ("expression" is often not even said, instead simply "this lambda here ...") in many programming languages is already an anonymous procedure and often an anonymous function. So saying "lambda function" makes it sound like "procedure function" or "function function" or "lambda lambda".