Hacker News Reader

Comment by VWWHFSfQ | Hacker News Reader

Provisioned concurrency is insanely expensive. If you have any kind of a thundering herd access pattern then Lambda is a complete non-starter because of the warm-up and scaling characteristics. We eventually just put an nginx/openresty server on a regular medium EC2 instance and got rid of Lambda from our stack completely and now we're paying about 1/300th the cost we were previously and the performance is infinitely better.

I'm sure it has some use-cases in some kind of backoffice task queue scenario, but Lambda is nearly unusable in a web context unless you have a very trivial amount of traffic.

View on HN

This is another example of AWS over marketing Lambda. Lambda is horrendously expensive when requests pass a certain level per second. You can graph it against ECS / EC2 to see the point it stops becoming economical.

Taking all of this into account, Lambda is then useful for a very small niche:

- Tasks that don't care about low P99 latency. These tend to be asynchronous processing workflows, as APIs in the customer request path tend to care about low P99 latency.

- Tasks that have a low request per second. Again, these tend to be asynchronous processing workflows.

You talk to anyone on the AWS serverless team and the conversation eventually focuses on toil. If you can quantify engineering toil for your organization, and give it a number, the point at which Lambda stops being economical shifts right, but it doesn't change the overall shape of the graph.

In my experience cold starts don't affect the p99 if you have substantial traffic, because you have enough lambdas consistently running that cold start rate is ~0.1%. P99.9 also matters though!

If you have substantial traffic, the cost savings of Lambda are gone, and you can just use ECS or something.

I use Netlify serverless functions (which is just a wrapper around AWS Lambda) because it basically fits the criteria for me. I have a low but bursty access pattern that fits into the free tier, and there's a static SPA page that can serve up instantly while the XHR triggers to do the cold start fetch. I don't think I would use it for anything consumer facing though. This is just a backend where an extra 300ms isn't going to make a big difference to the admins.

> This is another example of AWS over marketing Lambda. Lambda is horrendously expensive when requests pass a certain level per second.

I feel this is a gross misrepresentation of AWS Lambdas.

AWS lambdas are primarily tailored for background processes, event handlers, and infrequent invocations. This is how they are sold, including in AWS' serverless tutorials.

Even though they can scale like crazy, and even though you can put together an API with API Gateway or even Application Load Balancer, it's widely known that if your API handles more more traffic than a few requests per second then you're better off putting together your own service.

The rationale is that if you don't need to do much with a handler, or you don't expect to handle a lot of traffic on a small number of endpoints, AWS lambdas offer a cheaper solution to develop and operate. In some cases (most happy path cases?), they are actually free to use. Beyond a certain threshold, you're better off getting your own service to run on EC2/Fargate/ECS/whatever, specially given that once you have a service up and running then adding a controller is trivial.

> I feel this is a gross misrepresentation of AWS Lambdas.

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code as a ZIP file or container image, and Lambda automatically and precisely allocates compute execution power and runs your code based on the incoming request or event, for any scale of traffic. You can set up your code to automatically trigger from over 200 AWS services and SaaS applications or call it directly from any web or mobile app. You can write Lambda functions in your favorite language (Node.js, Python, Go, Java, and more) and use both serverless and container tools, such as AWS SAM or Docker CLI, to build, test, and deploy your functions.

https://aws.amazon.com/lambda/

Edit:

> it's widely known that if your API handles more more traffic than a few requests per second then you're better off putting together your own service.

How is it widely known? Is it on their documentation clearly or in their marketing materials to use another AWS product?

That's what's I mean by over marketing here. Requiring insider baseball knowledge because using it as described footguns your company at infection points isn't a great customer experience.

> AWS Lambda is a serverless compute service that lets you run code (...)

So? It can run your code the way you tell it to run, but you still need to have your head on your shoulders and know what you're doing, right?

> How is it widely known?

It's quite literally covered at the start of AWS's intro to serverless courses. Unless someone started hammering code without spending a minute learning about the technology or doing any reading at all whatsoever on the topic, this is immediately clear to everyone.

Let's put it differently: have you actually looked into AWS's docs on typical lamba usecases, lambda's pricing and lambda quotas?

> That's what's I mean by over marketing here. Requiring insider baseball knowledge (...)

This sort of stuff is covered quite literally in their marketing brochures. You need to even be completely detached from their marketing to not be aware of this. Let me be clear: you need to not have the faintest idea of what you are doing at all to be oblivious to this.

There's plenty of things to criticize AWD over, but I'm sorry but this requires complete ignorance and a complete lack of even the most cursory research to not be aware.

You've been going on and on. I linked you the AWS marketing page on Lambda that includes it scales with no infrastructure and can be used for all use case.

You've had two chances to cite something on their vast marketing and documentation other than marketing brochures (are you serious?) and AWS specific training, paid or otherwise.

You even quoted the wrong part of the marketing spiel.

Just upload your code as a ZIP file or container image, and Lambda automatically and precisely allocates compute execution power and runs your code based on the incoming request or event, for any scale of traffic

ANY scale of traffic, requests or events. Just upload a ZIP or image and you're done. We know that isn't the case, don't we? Even without AWS sales people showing up personally to provide us marketing brochures they wouldn't put on their website.

Insanely expensive is definitely a flexible term. I think numbers help here.

Provisions Concurrency $8.64 / GB / month

256 MB per Lambda (Assuming Python, Ruby, NodeJS, or Rust)

$2.16 per Lambda per month

A lot of organizations can probably make a good business case for keeping 100s or even 1000s of Lambda's warm. You also don't need to keep them warm 24x7, can get an additional 12% discount using savings plans, and if you're a big guy you get your EDP discount.

I'm sure there are plenty of companies that are happy to throw $5,000/mo at a problem that can be solved better for $250/mo. Not mine though.

Does your $250/mo include all the ops cost of other solutions?

Yes it does

You could make a fortune selling that by itself, unless your ops cost is just yolo provisioning and never doing backups/patching/etc.

I'll never understand how we got to this point of learned helplessness where people think hosted services like Lambda are the only ones capable of being secure and robust. It's madness..

That's not what I said or implied.

I'm not sure what you're trying to say then.

> your ops cost is just yolo provisioning and never doing backups/patching.

You think Amazon is the only one capable of doing backups and keeping software up to date?

No, but I think that it's super common to discount to $0 all the work that using lambda saves you from maintenance and operations. And if you can do any of that at scale for $250/mo you're lightyears ahead of nearly everyone.

You should look at offering this as a service perhaps. 2,500 250MB lambdas for $250/month with all AWS guarantees (ie, Multi-AZ, permissioning on every call etc etc) would be pretty compelling I think for folks running intermediate lambda workloads (ie, 5-10K lambadas at a time).

I'm not trying to offer it as a service. I'm trying to run my workload in a way that can scale from 0 -> 10,000 request/second in an instant and doesn't cost my company $5,000/month to do so.

It's pretty easy if you know what you're doing (or care to figure it out).

If you can do $250/month with all ops costs and features of lambda for 5,000 or 10,000 requests per second - you would be silly not to offer a service.

There are plenty of us who can run a system that scales to 10krps. That's relatively easy? I personally can't stand lambda and don't use it FWIW. I like EC2, I actually like fargate a lot of all sorts of things including lambda like services without a separate lambda for each request.

But for folks with a payload, that want the lambda like experience - if you have a solution, all ops cost included (ie, no well paid developer or ops person needed for customer) for $250/month for the scale we are talking here (2,500 x 250MB = 625GB etc) then you have an amazing solution going especially if you can do the networking, IAM controls etc that aws provides.

The problem I've seen, when folks say amazon is "insanely expensive" they are usually not actually comparing the AWS offering to a similar offering. If your cheap solution is not lambda like, you need to compare to EC2 or similar (with perhaps a good programmer doing something a bit more monolithic than aws).

it ends up being cheaper overall if you have high utilization of your provisioning since the per second fee while a function is running is cheaper. using https://calculator.aws/#/createCalculator/Lambda, if you have a steady 1 request/s and each requests takes 1 second, 2592000 seconds in a month. at 1024mb, i get 36.52 for provisioned and 43.72 for on demand. With autoscaling...you wont get 100% utilization, but it probably ends up being close enough to a wash

1 GB of lambda provisioned concurrency is $10.95 USD a month in us-east-1.

That's what you pay for your lambda sitting there doing nothing.

You can get an on-demand ec2 instance with 1 GB of RAM, for $9.13 USD a month, or $6.87 if you get a reserved instance.

You can fully utilize those instances the whole month.

Source: https://calculator.aws/#/estimate?id=7e1d1c2f32a2c63ba4ded19...

Yeah I used the ARM price. I should have pointed that out. It's definitely a tradeoff. Here's a few things I think are advantageous for Lambda,

* Integration with all the AWS Event Sources.

* Faster autoscaling.

* No need to setup a VPC (subnets, NAT gateway, security groups, VPC endpoints potentially, etc)

* No need to setup autoscaling, multi-AZ, etc.

* No need to support long lived instances.

I don't think one is always better than the other.

> I'm sure it has some use-cases in some kind of backoffice task queue scenario, but Lambda is nearly unusable in a web context unless you have a very trivial amount of traffic.

This has been the outcome for me on several projects too. Just use loadbalanced EC2 (or EB, for simplification) and pay for a few instances running 24/7. It's actually cheaper than having a busy lambda in all my cases.

The only other case (other than occasional backoffice jobs) would be long-tail stuff: an API endpoint that is used in rare situations: for example the "POST /datatakeout" or "DELETE /subscription/1337" or such. Things that might be heavy, require offbeat tools and so on. We've had them for building PDFs and .docx from reports; a feature used by <2% of the users, yet requiring all sorts of tools, from latex to pandoc.

I would guess this situation is maintained on purpose by AWS as the upsell reason for Fargate.

Yeah the caveats, gotchas, and workarounds you have to do to get something reasonable running on Lambda are just goofy.

At some point we just stopped and wondered why we were punishing ourselves with this stuff. We switched to a traditional webserver on regular EC2 instances and haven't looked back.

Have you run into issues with Lambda with complex tasks? I thought there was a 15 minute limit to tasks, plus a maximum storage size when importing large dependencies, etc?

The latex example did not run entirely on Lambda. Lambda would write a job into a queue (just Postgres), trigger a launch of a beefy ec2 instance, after which a worker on that ec2 picked up the job. Another lambda function would be called by the server itself to shut down the worker when all jobs were done.

Kludgy and slow. But it worked and did save some money, because the instance running this latex worker was big and chunky yet utilized maybe 10 hours a month.

Lambda was mostly acting as a kldugy load-balancer really.

...or a big latency budget? Slow start is fine for a sudden burst for a lot of use cases.

Reply on news.ycombinator.com