> if we looked in the opposite direction (where such light originates from) we would be seeing stars and galaxies at much more recent time (now)

If the preferred/non-preferred directions were "toward you" and "away from you", then rotating yourself to look in the "opposite direction" wouldn't make a difference.

(Note: not a physicist)