Sparrow: Distributed, Low Latency Scheduling (2013) [pdf]

From 2013. Doesn't seem like it was really updated.

https://github.com/radlab/sparrow?tab=readme-ov-file

Really great research work from Kay Ousterhout. Sparrow extends the "power of two choices" [1] to a more general formulation, and makes it really cheap to find very good choices (i.e., low-load servers) to run a task, without having to check the load on every server. As the number of tasks and number of servers grow, keeping an accurate picture of the load on every server makes the scheduler become a bottleneck. Kay's work showed a very interesting point in this design space for fast and high-quality scheduling.

Her PhD work went on to really understand the performance of complex distributed programs, like large Spark jobs, and a lot of at least earlier performance instrumentation in Spark is due to her.[2]

[1] https://brooker.co.za/blog/2012/01/17/two-random.html [2] http://kayousterhout.org/

This is a bit unrelated, but I've been looking into AWS Lambda or fly.io style scheduling recently. This specific response from and AWS Lambda engineer has been floating around in my brain: https://news.ycombinator.com/item?id=39501188

It seems like there is quite a bit of similarity between the type of scheduling that FAAS platforms need to do and the type of scheduling that Sparrow/Spark needs to do. Anyone know if these are truly related research spaces, or do they have enough differences that they can be viewed as distinct? Is it worth looking to data analytics frameworks for FAAS scheduling inspiration?

Another Matei Paper :)