GTX series cards don't fit in 1U/2U/3U rack servers very well due to the design of the coolers. The Tesla cards are available in passively cooled configuration (just a big heatsink) which allows them to be cooled using the existing fans in the chassis.

Facebook and Microsoft have their own 8-way GPU server designs that use a completely different connector for the GPU (SMX) instead of normal PCIe slots, again because of the motherboard/chassis design. (SMX is also used to expose the NVlink bus).

https://www.nextplatform.com/2017/03/17/open-hardware-pushes...

Companies like google prefer nvidia tesla data centre cards rather than the GTX 1080. Also, if I remember correctly nvidia k80 has two GPUs in one card along with 12 Gb per GPU for a total of 24 Gb. These servers have 8 nvidia K80 per system. A lot of compute power.

Using a Geforce card in a datacenter or "GPU cluster" setup voids the warranty, which might be a factor for them at scale.

The 'rules' for data centers are kinda different (also why the DGX-1 can't be used either) - cooling is very different and having ECC is a good thing.

It wouldn't surprise me if nVidia cuts favorable bulk deals on the K80, as well.