Keep in mind that on Earth-scale, movement through the fourth dimension would be pretty easy to notice. There's a lot of stuff on Earth, so seeing objects bounce in and out of view would be a pretty big deal.

But our universe as a whole is almost entirely empty space, with visible objects appearing to make up only about 5 percent of the universe's total volume. Noticing 4D things flit in and out of our 3D universe might be difficult at that scale.