Hacker News Reader

I just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to test, easy to debug, easy to maintain, not very expensive, and just kinda worked.

The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently running unless you keep track of it yourself. You'll only find out when you try to kick off a job and it get's throttled. I haven't contacted support yet to find out how hard it is for the limit to get raised or what they'll raise it to.

We had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.

Yeah, most of the limits are there just to prevent "whoops, we fired up 500 $2/hour servers and left them running all month, can you refund us?" sort of newbie situations. Very few are technological and AWS has been happy to bump them up for me in each occasion I've had a need.

I've had plenty of cases where it's taken time, and required escalation and justification to get even relatively modest instance limits raised, depending on the data centre and instance types.

It's not technological - it's about ensuring they have a reasonable handle on capacity and what they happen to have plenty of.

It's not a problem as long as you're not asking for astronomical limits, but people need to be aware of it and request increases in advance of starting to hit limits rather than assume it'll be near immediate to get them lifted.

Some are, some aren't. For example, my experience has been that it's harder to get the 100 S3 bucket-per-account limit raised.

That used to be a hard technological limitation but they fixed it a few months back. You should be able to get more now (but you have to have a good justification for it).

100 seems an odd number to be a technological limitation, doesn't it? It's not a power of two or anything so it's not the size of an integer. How could technology have been limiting it?

S3 bucket names are globally scoped - ie. they are literally scarce resources.

Technology in the sense that it was a hard coded variable, but it was picked arbitrarily.

What would be a good justification? Is there anything you can do with 101 buckets that you can't do with folders one bucket?

Requester-pays billing (http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketBilling...) for more than a hundred clients, static sites served through CloudFront, etc.

IIRC that was one of the few the AWS docs specifically called out as unraisable.

I once wanted to raise the limit of g2 instances from 2 to 100 in us-east-1 and the reply from the customer agent was he first "has to check if that does not affect overall region stability" WTF Anyway, it was raised a couple of days later.

Very useful, thanks!

raising the limit is a pretty easy request to aws support.

So google compute setup I did a while back with preemptible instances + a celery queue + some autoscale based on load...

The guts to make all that work was 50 or so lines of config. I think my auto scale script was 20 lines or so of Python.

I guess the biggest downside was spinning up the new server took about 2 minutes, so for big load spikes it took a bit for it to level out... but with GCE per minute billing, all the napkin math I did says it is a fair bit cheaper per cpu unit to do it this way.

In conclusion, someone good with devops + a normal work queue system could do this years ago. I guess it's cool to lower the barrier to entry, but not it does not seem like a game changer. It totally IS cool to scale out backend work to a job farm. Just seems like not the only way to do it.

That's essentially what Google Cloud Dataproc gives you (managed Hadoop/Spark):

- Per-minute billing

- 0-to-cluster in under 90 seconds (aim for 30 seconds)

- Pre-emptiblem VMs

- Custom VMs

Now you start with a job, pay a 30 second penalty, and execute it on an entirely ephemeral cluster. The "get a cluster and fill it with jobs and round up to an hour" model is indeed outdated IMHO.

(work at Google Cloud)

Well it depends on my workload. The setup above on GCE had a stable load of X thousand jobs per minute, then burst loads up to 100x for short times. So for me it made sense to have a 24/7 celery cluster for the base load and add and remove nodes for the variable node. There was never a point shutting down the cluster made sense.

" I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images."

Because those are the things that need to be done in the real world, 99% of the time.

Sadly, we don't all get to work on cool stuff :)

But I think as more people get access to the power and find uses cases, it will get used more.

The 'next' step for Lambda is serverless hosting for web-sites, something which Lambda is 'almost but not quite' ready for, the limiting factor being transactional latency, especially in some weird cases. Once they have that sorted out, I will never want to see a server again :)

Except for the fact Lambda is limited to 100 concurrent executions.

This isn't a fact. The default limit is 100 concurrent per region.

Also a simple conversation with AWS increases the limit[0].

They obviously don't want people to write absurd runaway structures and 100 is an arbitrarily chosen value to guard against that.

[0]: https://aws.amazon.com/lambda/faqs/

>Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure.

And you've spent the cost of building out that 8 server infrastructure in one batch.

One of the nice things about lambda is the billing is super granular - you get billed at 100ms intervals.

Assuming 70 hours at $0.000000834/100ms [1]

The whole job costs $2.10.

[1] - https://aws.amazon.com/lambda/pricing/

By comparison (since I was curious), 70 hours on an m3.medium spot instance will run around $0.70. On an on-demand, it's about $5.39. EMR will cost you about $7.00 on top of the EC2 costs.

If you can peg the CPU and don't mind getting interrupted, spot instances are still a fair bit cheaper. But Lambda looks pretty attractive for any other use-case where the statelessness of Lambda doesn't bite you.

Keep in mind that with EC2 you're billed hourly so the fastest a 70 cpu-hour job could finish on m3.medium for $0.70 is 1 hour, and that's ignoring setup time, etc.

Meanwhile, on Lambda, you can actually run 1600 60s jobs (or 27 CPU-hours) in 3 minutes. This is inclusive of setup time, job submission, stragglers, etc. [1]

Of course, if you've got sustained load, it's cheaper to go with spot instances, but the "occasionally I need a buttload of compute," model is well-served by Lambda.

[1] http://ericjonas.com/pywren.html

As a note for people, if your constraints are a bit different then these are some services to check out:

Joyent Manta: https://www.joyent.com/manta

Hyper: http://hyper.sh

Possibly Joyent Triton: https://www.joyent.com/triton

I personally often want to run a bunch of things for ~1-15 minutes, and have too much data or setup to fit neatly in a lambda function. However, I don't need 1000 things running simultaneously, although manta would help still there.

I'd love to see some more layers over the top of services like this, hopefully someday getting us back to picloud. I miss that service.

How secure are docker hosts like hyper.sh? I've always been skeptical, the multitenant docker security story hasn't been very encouraging, or has that changed?

hyper.sh containers are kernel-isolated like virtual machines.

70 * 60 * 60 * 10 * $0.000000834 = $2.1

As the other poster who replied to you showed, it's actually incredibly cheap. Additionally, it's perfectly elastic. That expensive hardware you provision or buy for your server that can handle this task (at a much, MUCH slower rate) would cost you significantly more. In addition to the cost of the server and running it, you also have a much harder job developing it. If we're talking about multicore, you're now managing the job concurrently on a single machine, and need the code to facilitate it.

I also feel like your comment is implying that the outcomes of the lambda option vs the worker server are comparable. As mentioned, the worker machine will still run in hours, rather than less than a second.

The cost to execute a 250ms Lambda on the most expensive tier 1M times is about $7.70. If your data is in S3 the cost of 1M GETs at the most expensive is $0.40. You don't pay for transfer between S3 and Lambda.

if it's one of you will not be able to get the limit if you do this even at a rate of 1 req/sec you will be paying close to 1.4 million USD per month just for compute.