That doesn't mean the external physical world doesn't exist, the information we use to construct that model must come from somewhere, and we can deduce that the source is a persistent and consistent one.
The philosopher Husserl said: “The tree plain and simple, the thing of nature, is as different as it can be from this perceived tree as such, which as perceptual meaning belongs to the perception, and that inseparably.”
He came up with the idea of the noema which is our experience of something, and noesis which is our conscious act of perception. For me, that's our act of interpretation of our sensory perceptions. Sometimes this all goes wrong and we construct a flawed model that does not correspond perfectly to actual external reality, such as when we are deceived by optical illusions, stage magic or just hallucinate. Fortunately we can test and correct our perceptions through action in the physical world.
I'm an out-and-out physicalist but I think he is quite correct, we must distinguish between our internal perception of things and how things actually are. Fortunately science is extremely powerful in this regard. It has allowed us to decouple our model of the world from the limitations of our perceptual system, and come up with rigorous models of reality such as Relativity and Quantum Mechanics that are not tied to direct interpretation by our perceptive systems.