It's doesn't quite work that way.

Oversimplifying it a bit. Imagine a swimmer swims 5m/s and the current adds/subtracts 1m/s. The pool is 50m. No current each length should take 10s. Going with the current takes 8.3s and going against the current takes 12.5 seconds. 100m takes 20.8s instead of 20s. Why?

While the benefit/detriment ratio is 50-50 in terms of distance, it's not 50-50 in terms of time. You spend less time benefiting from the current because you are going faster.

Alternatively, if the race was a fixed 10seconds in each direction you would go 60m and 40m, making it equivalent.