Technically speaking, we're talking about four dimensional space. It doesn't really make sense to call such a space "The Fourth Dimension", any more than real life space is "The Third Dimension", or a tabletop is "The Second Dimension". This sometimes trips people up into arguing over whether The Fourth Dimension "is" time, or whatever. For that matter, maybe the first dimension is time, and the second, third, and fourth dimensions are space. These things aren't ordered, and in fact you can't really distinguish between the three familiar spatial dimensions: imagine trying to point along dimension one, whatever that means.
The familiar three-dimensional space as we know it is three dimensional because you can put three straight lines to meet at right angles to each other, and no more. And you can label those lines x, y, and z if you like and pick their orientation. Four dimensional space allows you to cram another in. Two dimensional space only allows two, and one dimensional space is just a single line.