I'm a Dask evangelist. It's a remarkable tool and is one of the first I reach for when this problem arises. Maybe it's not well known?

I mean, don't reach for Spark. Just get Dask, a databased or R's disk.frame is mostly fine.