My current thinking is that it would make sense to implement the standard DataFrame abstraction (with all the generic functionality it offers) with a couple kinds of distributed data frame. One approach is to make a distributed stack of local data frames; another is to make a single data frame where each column is a distributed vector. Not sure which is better. Fortunately, Julia makes it really easy to experiment with these kinds of things. Of course, as soon as you have a system like that you need a scheduler for work at which point you want something like Dask or Dagger (https://github.com/JuliaParallel/Dagger.jl).

The basic Dask-like functionality is in Dagger.jl, we have plans to move out the distributed array functionality out of it and keep just the scheduler in there.

I'm working on a set of functions to work with many DataFrames (or anything) in parallel and do it out of core if possible, it's basically like JuliaDB but with a FileTree abstraction rather than a table abstraction.

http://shashi.biz/FileTrees.jl/lazy-parallel/

Would Query.jl[1] be an alternative to consider?

Their README says:

The package currently provides working implementations for in-memory data sources, but will eventually be able to translate queries into e.g. SQL. There is a prototype implementation of such a "query provider" for SQLite in the package, but it is experimental at this point and only works for a very small subset of queries.

Still early days, but sounds like they're working on the same problem.

[1] https://github.com/queryverse/Query.jl