Big data sorting is a very different problem. At the top level, a merge sort is mandatory, because that's the only algorithm that can be effectively parallelized across N nodes.
"Merge sort is the only algorithm that can effectively be parallelized" is not a good explanation for the lack of quick sort usage in the sort benchmark. In a parallel sorting system, you can decouple the merging step from the sorting of partial runs. That is, you'd still want the fastest algorithm for sorting partial runs, and then implement a fast merge operation.
So what's the reason? I can only offer speculations based on my own experience.
Contrary to what the name suggests, the sort benchmark is actually not designed primarily to test the in-memory sorting part of the system. It is designed to stress the entire system. When you are required to sort 100TB, or even PBs+ of data, your software system's ability to optimize for I/O (including reading from/writing to disks, sending data across the network, pipelining) is far more important than the actual sorting itself. Of course, you shouldn't have a very slow sort algorithm either.
The reason I used TimSort was because it was relatively fast, already implemented in Apache Spark, I was really familiar with it, so I could tweak it to do what I needed. The vast majority of the time I spent in the sorting part was to tune memory layout (to make sure there's 0 JVM garbage collection, and good cache locality), and to remove any virtual function calls. Once tuned, the sorting time was a very small fraction of the overall time. Using TimSort (which can merge partially sorted runs very quickly) also allowed me to implement just one highly optimized sorting algorithm, rather than having to implement one algorithm for sorting the partial runs, and another algorithm for merging them.
The sort benchmark also defines the key space (10 byte key) as fixed length. My guess is that the fastest in-memory sorting algorithm for this case would be a highly optimized radix sort, which the 2016 Gray sort record used.
It is one of the rare algorithms that is equally good when run by a computer and by hand.
For those aren't familiar with sleepsort, it looks something like this, where you use whatever concurrency is available in your programming language of choice to execute the for-loop in parallel
for n in array:
sleep(n)
print(n)> It's very intuitive and simple, at least to me ?
Once you understand mergesort, you have a eureka moment and it all makes sense. But I've never met anyone who thought merge sort was intuitive and simple - especially when learning it the first time.
Also, for future readers, this wasn't a flex, it was intended as an observation of weird things about brains, I learned many sorting algo (like many readers here) and only merge sort kinda stuck.
It just streams the data back and forth until it's sorted.
There are many - this has been an area of study since CS began. Have a look at vol 3 of Knuth, for starters.
Also, https://en.wikipedia.org/wiki/External_sorting talks about distribution sort; there are others.
Before starting I decided to figure out why the cross reference was taking so long. Each reference was added to a dynamic memory array, until the memory was filled and then the references were written to a disk file instead. At the end of the assembly, the array was sorted with a Bubble sort. Then for each symbol in the symbol table, the table and file were read linearly to find each reference for that symbol. This was an O(n^2) process - no wonder it was so slow!
I took out the Bubble sort and did a Merge sort on the file, then read the file to create the cross reference directly since it was in the proper order. When the time went from 30 minutes to 30 seconds, suddenly it wasn't important to have an option to turn it off anymore.