That is, in deep space almost nothing is happening while here on earth a lot is happening to increase entropy, so why does time go at the same speed in both places?
One could imagine that beyond some cosmological horizon the rules of physics as we know them break down and there is no zero point energy.
Normally you think of the rules of physics breaking down at a singularity point, say, beyond a black hole's event horizon or at the start of the big bang. If you subscribe to the idea that the big bang event shaped the laws of physics we know, then beyond the horizon of the "expanding" big bang, those laws won't necessarily apply.
I put expanding in quotes because I subscribe to the idea of a holographic universe. I like to think that while the universe may appear to be expanding at an accelerated pace to us, you can also think of our universe as shrinking within a fixed boundary. (Fixed relative to what?)
For example, imagine a 2D holographic plate (our true plane of existence), through which you can view an image of a 3D virtual object (our perceived plane of existance). Now imagine the interference pattern on that plate is dynamic and increases in entropy, defining and arrow of time for the objects in the 3D virtual image. The plate doesn't change size, but the complexity of the interference pattern increases, creating increasingly finer and finer details, such that from the perspective of a virtual object, everything appears to be expanding, but from an outside observer's perspective the holographic image appears to be shrinking as the entropy increases.
Obviously if something like a human brain was there to observe things, you'd have the entropy of your neurons to be sure of the direction as well. =D