Have you considered using HDFS?

There is a fairly informative post here: https://www.quora.com/How-are-most-data-sets-for-large-scale...