There's a lot of room between a laptop with 16GB of RAM and a "Big Data cluster" - in my experience the easiest solution is to spin up a VM in GCP and crank the RAM way up.

It blew my mind when I realized I could add / remove RAM by turning off the instance, dragging a slider, then turning it back on.

I also find h5py really useful for creating massive numpy arrays on disk that are too large to fit in memory. I used it to precompute CNN features for a video classification model (much faster than computing on each gradient descent pass) and it makes it easy to read/write parts of a numpy array when the entire numpy array is too big to fit in memory.