Sampling or subetting your data is almost always the correct response, especially for analysis.

If you're literally processing TB or PB of data, you'll want to parallelise. Though shell tools do this amazingly well.