The question of time is deeper than this. Why is it that your brain or for that matter any physical system is dependent on it's previous configurations? Why is it that there is such a thing as previous? The mathematical descriptions of reality we have devised are symmetrical in time...

I can imagine a universe where alternate number like systems are more appropriate as heuristics for day to day living. Why should every universe be easily described through isomorphisms to the infinite cyclic group? Why could a universe not be more easily understood (compressed) within the minds of it's inhabitants with systems which are non-abelian groups? Perhaps you could say that there can exist no universes where intelligent beings can exist which contain aspects which are compressible using such systems. But you would have to prove it.

I think the arrow of time question is very interesting, I would love to see and understand an answer to this before I die.