That's what Spark is for. You can do petabyte-scale jobs... with DataFrames.