Hacker News Reader

Comment by teej | Hacker News Reader

I just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to test, easy to debug, easy to maintain, not very expensive, and just kinda worked.

The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently running unless you keep track of it yourself. You'll only find out when you try to kick off a job and it get's throttled. I haven't contacted support yet to find out how hard it is for the limit to get raised or what they'll raise it to.

View on HN

We had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.

Yeah, most of the limits are there just to prevent "whoops, we fired up 500 $2/hour servers and left them running all month, can you refund us?" sort of newbie situations. Very few are technological and AWS has been happy to bump them up for me in each occasion I've had a need.

I've had plenty of cases where it's taken time, and required escalation and justification to get even relatively modest instance limits raised, depending on the data centre and instance types.

It's not technological - it's about ensuring they have a reasonable handle on capacity and what they happen to have plenty of.

It's not a problem as long as you're not asking for astronomical limits, but people need to be aware of it and request increases in advance of starting to hit limits rather than assume it'll be near immediate to get them lifted.

Some are, some aren't. For example, my experience has been that it's harder to get the 100 S3 bucket-per-account limit raised.

That used to be a hard technological limitation but they fixed it a few months back. You should be able to get more now (but you have to have a good justification for it).

100 seems an odd number to be a technological limitation, doesn't it? It's not a power of two or anything so it's not the size of an integer. How could technology have been limiting it?

S3 bucket names are globally scoped - ie. they are literally scarce resources.

Technology in the sense that it was a hard coded variable, but it was picked arbitrarily.

What would be a good justification? Is there anything you can do with 101 buckets that you can't do with folders one bucket?

Requester-pays billing (http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketBilling...) for more than a hundred clients, static sites served through CloudFront, etc.

IIRC that was one of the few the AWS docs specifically called out as unraisable.

I once wanted to raise the limit of g2 instances from 2 to 100 in us-east-1 and the reply from the customer agent was he first "has to check if that does not affect overall region stability" WTF Anyway, it was raised a couple of days later.

Very useful, thanks!

raising the limit is a pretty easy request to aws support.

Reply on news.ycombinator.com