Seems like Tesla could really benefit from this about to be released optimizer that reduces intra-GPU communication [0].

[0] https://github.com/NousResearch/DisTrO/blob/main/A_Prelimina...