Running Arbitrary Executables in AWS Lambda

Hi Tim,

1. What about support for cron-like periodic events? Maybe something like cron.yaml in Elastic Beanstalk, but serverless.

2. Does Java support means, that it will be possible to write lambdas in any JVM hosted language, like for example Clojure?

Python (must have). And then Ruby, Lua, R, Go, and C# in no particular order (just suggestions).

R would be huge for statistical computing, which Lambda holds a lot of potential for -- research would never be the same. Lua would be fitting/appealing just because of its speed and small runtime. The others seem to hold relatively happy spots on TIOBE and in Stack Overflow tags and would fit the Lambda model well.

There are a bunch of other events it can read from now: https://aws.amazon.com/blogs/aws/aws-lambda-update-productio...

There's also the dynamodb streams preview: http://docs.aws.amazon.com/amazondynamodb/latest/developergu...

I think the future will probably be measured in GB/s.

I recently tried WebJobs and found it to be simply awesome. The best aspects I noticed are:

* Seamlessness - it has an in-built processing pipeline (using queues, service bus etc.), whereas with Lambda you'll have to do a little more work to connect the components e.g. S3, SNS.

* Scalability (although both AWS and Azure offer auto-scaling) - Lambda has an execution timeout of 60 seconds for functions. If the functions are I/O bound then WebJobs is more suitable, else Azure Batch offers better CPU bound work support.

* Ease of deployment - 'drop and use', which I don't think Lambda offers out-of-the-box because there seems to be dependency on Node.js?! The WebJobs dashboard under Azure Portal is a great way to track invocations and logs.

* Cost - WebJobs comes free with App Services (Web Apps)

The one and only shortcoming of WebJobs seems be the lack of recurring tasks (but there's a workaround by adding the job back to the queue on its completion).

Further reading:

http://www.troyhunt.com/2015/01/azure-webjobs-are-awesome-an...

http://www.troyhunt.com/2015/04/orchestrating-massive-parall...

http://www.hanselman.com/blog/IntroducingWindowsAzureWebJobs...

An on-premise alternative can be Hangfire, Sidekiq or Quartz.

I thought WebJobs had more of a batch/queue processing focus, whereas Lambda is realtime?

Linux userland binary compatibility in general is surprisingly fragmented, thanks to the introduction of so symbol versions about a decade ago.

http://harmful.cat-v.org/software/dynamic-linking/versioned-...

https://refspecs.linuxfoundation.org/LSB_3.1.0/LSB-Core-gene...

I don't believe Amazon Linux is any worse than other Linux distributions in this regard. If you build a binary on an older Linux distribution (like CentOS 5), then there's a good chance it will run on most new distributions. The opposite is unfortunately not true.

Also, some libraries, like libcurl, don't seem to have consistent symbol versions across distributions. If your application links against libcurl, chances are you won't be able to create a single binary that runs on very many distributions.

I compiled a Golang static executable on Ubuntu uploaded it to AWS Lambda (which is Amazon Linux) and it works.

Didn't tried with C/C++.

From an executable standpoint there's not a ton of lockin -- anything running a certain version of libc should behave the same way. You can compile an executable that will work on both Ubuntu and Amazon Linux.

You could run them under Wine, or use C#...

Tldr: Only if your events are extremely infrequent. With an inter-event arrival rate of seconds to few minutes, your jvm processes get frozen and thawed out the vast majority of the time to avoid the initialization/load overhead that would otherwise result.

See https://aws.amazon.com/blogs/compute/container-reuse-in-lamb... for a more complete description.