Physics (as far as we know) doesn't care about the direction of time. Cause and effect depends on a fixed direction of time.
Math may not care (and it's true that most of the fundamental physics equations are often time reversible). But the universe (and any experiment to date) clearly shows that time has a direction and the equations work in that direction.
Charge, Parity, and Time Reversal Symmetry is a fundamental symmetry of physical laws under the simultaneous transformations of charge conjugation (C), parity transformation (P), and time reversal (T). CPT is the only combination of C, P and T that is observed to be an exact symmetry of nature at the fundamental level. The CPT theorem says that CPT symmetry holds for all physical phenomena, or more precisely, that any Lorentz invariant local quantum field theory with a Hermitian Hamiltonian must have CPT symmetry.