It seems space could be at most a 6 x 6 x 6 lattice, maybe forming a hyper-torus or something like that. And time, only 6 possible points in time, too. Hopefully nobody starts walking around in that tiny universe and discovers that there are more than 5 distinct places to visit. To really get rid of 6, you probably have to limit the entire universe to 6 states, but even then I am not really confident that 6 remains out of reach.
Note that there is also a tension between ordinals and cardinals, you can number points in each dimension of the lattice universe with 0 to 5 but that are of course 6 numbers. But I decided to ignore that, it just makes the idea even harder to realize.