AWS Batch – Fully Managed Batch Processing | Hacker News Reader

AWS Batch – Fully Managed Batch Processing

aws.amazon.com

125 pointsirs24 comments

We have been using this architecture for containerized batch processing for more than a year:

http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-d...

It is very convenient to be able to define any jobs as Docker containers, using the best language for the task, and define dependencies between their inputs and output explicitly, not being tied to any particular paradigm like MapReduce.

I am more than happy to let AWS handle plumbing for this instead of having to maintain and operate our custom solution.

By chance, have you tried https://gc3-uzh-ch.github.io/elasticluster/ ?

Also, have you looked at Airflow?

Not comparing them, just curious.

First time I hear about Elasticluster. Regarding cluster provisioning, auto-scaling is critical for cost and convenience. We managed to do this with a simple job queue that outputs a CloudWatch metric that in turn controls an auto-scaling group of spot instances. Works like a charm but AWS Batch should make this redundant.

We took a look at Airflow, which looks nice, but we haven't had a compelling reason not to use Luigi.

We've looked at Airflow. Two main things why we went with Luigi are

* time-based and dependency-based scheduling do not mix well in our experience, if you try to do both at the same time you end up with a very complicated execution model. Luigi is much simpler, it is basically a makefile. To handle time one can just add dummy dependencies where "exists()" function returns true if now()>X, and/or just run the whole thing by cron.

* We heavily use dynamic dependencies in Luigi, where one job decides how many childs to spawn and what parameters to use at run time. Makes it easy to do e.g. mapreduce style jobs purely in Luigi.

AdRoll has done some very cool stuff using containers for data processing. Afaik they were some of the first people to use this approach, and I think it's worked out quite well for them and given them quite an edge over their competitors.

If you're interested in building this type of infrastructure for yourself, I'd recommend taking a look at http://github.com/pachyderm/pachyderm. I'm one of the founders so this is a somewhat shameless plug, but Pachyderm is designed to get people most of the way to what AdRoll has built without having to build as much of the "plumbing" from scratch. AdRoll was some of our earliest inspiration for the design of Pachyderm.

Thanks! Pachyderm's version control over datasets is a killer feature.

"AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted."

Optimal is a pretty strong claim, given that this is an NP-Hard problem. Or do we just toss the word "optimal" around now?

You know what they meant

How do you know that AWS Batch did not prove N=NP?

Maybe that is the secret sauce.

I could not tell from the marketing literature - what's the difference between this and EMR?

I can only assume, this sits higher than EMR. EMR is a managed Hadoop (and likes). This talks about running frameworks like luigi which is a workflow engine for, for example, Hadoop jobs.

The jobs management interface. This is more like SLURM, Torque, LSF, etc...

It looks like you submit jobs to a queue and it spins up capacity to execute them. In that sense it's like EMR. However the jobs are a Docker image or a zip file with shell scripts. In that sense it's not like EMR.

I haven't gotten far enough yet to see if it has any distributed compute support, or if each job runs on a single machine. EMR is designed for master/worker architectures with many workers. It looks like AWS Batch might be designed to run each job on a single machine.

It's more like the container service without setting up your own cluster

If this is what I think it is, I may have finally found my Heroku Scheduler killer.

This seems like a step backwards to me in some ways. I'd prefer to see them evolve Lambda to support containers, longer jobs, and better workflows instead. I thought we were moving away from EC2 with its slow provisioning, spot bidding, and per-hour billing.

I think there is a useful distinction and place for this.

Lambda is for shared-compute. You don't need a dedicated server to run a "function" that takes < 60 seconds and can be called in a stateless manner as an API endpoint.

This is dedicated host compute-heavy batch processing. It's a pain to do this at scale!

I've built systems for running large scale life science embarrassingly parallelizable problems on EC2 and wish I had something like this!

Imagine you have a set of input S3 files, each needs multiple-hours of compute to produce output S3 files. Doesn't seem that hard, until EC2 instances fail, programs crash, etc. etc.

And the idea here is that you have a large parallelizeable problem but you don't need the Hadoop ecosystem? To put it another way, why not just use EMR?

Yes, no Hadoop.

Quite a lot of life science work is stand-alone programs that are domain-specific and read and write flat-files.

Also every legacy mainframe Cobol batch system on the planet.

Will AWS Batch support the running of COBOL batch programs?

This. As someone who works on bioinformatics workflows one of the more difficult aspects of my job is to try to explain to other software folks why we do what we do. While scheduling is a solved problem the issue is that you're dealing with a ton of command line tools expecting POSIX filesystems and often not involving parallelization

What is the selling point of this? Poor man's Map(no reduce)??

For people involved with scientific workflow engines this is the Amazon equivalent of the Google Genomics pipelines API. For most of the computational world it's silly but for us it is pretty important.

Reply on news.ycombinator.com