I find this an odd concept, "learning the arrow of time". The very act of learning presumes an arrow of time already exists: one learns when one's knowledge at time t_2 is greater than one's knowledge at time t_1, where t_2 > t_1. If we remove the requirement that t_2 > t_1, then we cannot distinguish the act of learning something from the act of forgetting something we used to know.
"Learning" here means averaging over many rollouts (i.e. simulation runs) of a policy. Naturally, time only increases in those runs, as you require.