These systems are all massive clusters. On the compute nodes they generally run a lightweight kernel that provides minimal services--just what is needed to run the payload application and support things like communication with other nodes. That pretty much rules out the closed source Unix systems and normal versions of Windows for the compute nodes, as they would include all kinds of things you don't need that would take up resources that you'd rather have available for computing. If you aren't going to develop your own lightweight kernel for your compute nodes, you need an open source kernel to strip down. So that basically brings it down to Linux and
BSD.I'd guess Linux is dominant because it is more popular than BSD on servers and desktops, so the people who developed the kernels for the compute nodes were simply more likely to be familiar with Linux.