This is true from the standpoint of materialism but not necessarily fundamentally true.
How do you know you have a brain? As you explore this question, you’ll realize that the knowledge that you have a brain only manifests as appearances within consciousness.
It’s not necessary true that these appearances are giving you a window into an objective material universe. Instead it might be possible that your consciousness is a product of a simulation where your entire subjectivity - including the observation that you have a brain - is a manifestation of another mechanism that is outside of observability.
The point is that we simply don’t know what’s at rock bottom - an objective universe, a simulation, or an alien’s dream. Therefore the “arrow” of causality might flow from consciousness towards material as opposed to the other way around.