The point is that we don't see eggs break at some times and reassemble themselves at others. Whether time is flowing "forwards" or "backwards" is just a label - if you decided to start calling "tomorrow" "yesterday", it wouldn't change anything except what you say.

However, if you could break an egg and then unbreak it, that would be remarkable (and observable), whether you did it tomorrow or yesterday.