Another idea: many of the swimmers breathe only to one side (in the races that they breathe at all), so in one direction they would be able to (better) see the faster swimmers towards the center. Another possible psychological factor.