Thanks for the tip, I didn't think to copy the data to ephemeral storage like that. That'll probably speed things up a lot.

I ended up splitting the data in to a relatively small number (~200) of ~30MB gzipped files in order to initially saturate the mappers and speed things up. If that's not necessary after moving to ephemeral storage that's fine by me!