Fugue is a layer to abstract out these distributed computation backends, and it looks like a nice programming interface.
Does Fugue take advantage of each sublayer that already uses Arrow?
First, present day users of Python need to understand how and why Python came under the spotlight. There was always a standoff between programmers who created unimaginative huge programs full of drool and red-taping, and programmers who wanted larger freedom of expression, less strings attached. The later group was usually the more savvy ones.
In a way very similar to how an art student might be spending months studying a model, using a whole bunch of pencils starting from 10B and ending in 10H, various chalks, coal sticks and so on... and would still produce a... "study of a model #80907", which is ugly, anatomically incorrect and just boring. And there's an accomplished artist who can just stick her finger into the chimney, grab some soot, and in a matter of minutes make a great drawing, which will be lively, expressive, you name it.
So... Python, and Perl before it were the soot. The junk languages a more experienced programmer would go to just to show those boring Java programmers "how it's done". But, Python programmers who came in the next wave thought that soot is the good tool to learn how to make good drawings. And, today, we have academies full of students trying very hard to draw models with materials and instruments which are very inappropriate for the task. (Unless you know anything about art education today, the example isn't that big of a stretch of what happened in it around 70s-80s.)
---
I don't care if Python is a glue code for Scala or C or Rust: it doesn't matter. Python, as a language, is inadequate for dealing with concurrency. It needs to remove a bunch of stuff before it can start adding stuff that can be used to that end. It's a language with a lot of mutation semantics which are hard to interpret / implement correctly (what would that even mean?) in distributed context. It's a language with a lot of implicit stuff going on that is somewhat useful (but is not useful enough) if you want to have a quick and dirty "sketch" quality code, but will be devastating in distributed context.
Things like decorators, context managers, imperative loops with break and continue, error handing mechanism, threading -- all of this must go before Python can start to think about becoming a decent language for distributed systems. But, probably more: I would need to research this in much depth to tell for sure if things like method calls would work well for example.
It's a waste of time to try today to fit Python into distributed computation. You will either have to put a humongous effort purging a better half of the language (making all of the famed support libraries useless), or you will end up with a defective hodge-podge mess (which is all those famed support libraries are, including those which aim to do distributed programming in Python).
A much better distributed platform is Erlang. When comparing it to something like Slurm, the benefit is that the programmer has superior access to sharing information between workers, the programmer has programmatic (and superior) control for choosing the workers.
The problem with people in WLM world (that's where Slurm, MPI, OpenMP and friends come from) is that they have never crossed paths with Erlang. They don't even know what they don't know. A lot of the stuff that has to do with the "distributed" part of their computations is perceived outside of their control, something outsourced to infra people: they just need to put a "machine file" somewhere, or use some HTML dashboard to select the nodes where their program needs to run, and hope for the best.
Also, the WLM world has a very peculiar take on distributed programming: they need it to run batches of jobs. Jobs start, finish, and, hopefully, produce results. They don't need to build interactive systems, or deal with upgrades, or create databases etc. So, it's just not an answer to more general problems associated with distributed programming.