We wrote our own scheduler, called Quentin, which we may opensource. Essentially it is a simple queue that takes care of running containers and managing pools of instances that containers run on by integrating with AWS autoscaling.

It is a very straightforward service, for example it doesn't do retries (done by Luigi), it is not highly available and not distributed. That makes things much easier to maintain, and we don't need HA since jobs inputs/outputs are in S3 anyway. Luigi makes it easy to restart parts of the pipeline if scheduler ever goes down.