Hacker News Reader

It looks like GNU sort already does this internally, but with smarter temp file management. Increasing the memory size looks like a good idea, and if you have multiple hard drives, it may help to specify a temporary directory on a different hard drive with the -T option, since this problem looks IO-bound.

View on HN

I believe GNU sort is a single-threaded external multi-pass/multi-way mergesort.

http://en.wikipedia.org/wiki/External_sorting

The reason to split is not to replicate mergesort, but rather to get around single threading in GNU sort. Of course, there are better options if this is a regular task, but GNU sort just happens to be really common and easy if it's a one-off.

I do wish I had a better split (since split seems to read the whole file) and a better sort (that was parallel and reasonably common) though.

Reply on news.ycombinator.com