As long as you can map time to a number line, it's a valid representation. We just happen to have hardware acceleration for 3-dimensions, and the 4th is just completely unintuitive to us.
When it comes to geometry and not just vector spaces, time dimensions have a different definition of distance than do space dimensions. There's a minus in the formula where you would usually have a plus. And this means that shapes in this space behave very differently than what we're after when imagining a hypercube or hypersphere, for example.
We want to think of a 4 dimensional space where all the dimensions are indistinguishable, but the minus sign in the metric distinctly identifies the time dimension. For this reason, physicists typically call this kind of space a 3+1 dimensional space rather than a 4 dimensional one.