That's the first order truth. If you don't have any knowledge about the system, you can't infer causality with observations alone.

With some generic assumptions, or prior knowledge about the system, you can do causal discovery.

For example, just the assumption that there is additive random noise enables discovering causal arrows just by observing the system.