The general reason comes down to inter-node bandwidth. Things like Google's search or SETI are able to use standard interconnect (or the open internet in SETI's case) because the problem, link counting and signal analysis, can be broken down into individual pieces that don't interact very much between sections of the computation.

Things like physics simulations, weather etc., have a lot of interaction between any divisions you could try to draw to be able to split the information between nodes. To work with these problems you need a faster interconnect between processors than ethernet provides, infiniband seems a popular choice but I'm no expert on the details of their architecture. These problems also need to move in relative lockstep meaning loosely connected systems don't work as well or at least don't provide any real advantages. Node homogeneity makes these tightly coupled processing nodes easier to manage.

Occasionally there will be upgrades to a super computer but generally they're run 24/7 until they're too far behind the technology, both in speed and lower power consumption, that it becomes cheaper to completely replace it than to try to do an upgrade that will replace more than half the cluster's total electronics to begin with.