The operative phrase here being "you don't see". Of course, seeing is a matter of perception and perspective. It could very well be that time is flowing "backwards" and eggs are in fact reassembling themselves, but we are just perceiving time in reverse.
You know, when I hear physicists try to tackle philosophical problems, it's usually kind of embarrassing. These guys are rarely trained in philosophy and are out of their depth.
As Feinman famously said to his own students, "Shut up and calculate."
I wish more physicists would follow his advice, and leave philosophy to philosophers.